Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing Misc/73527
Full headers

From: sav2@le.ac.uk
Subject: GTI extensions
Compose reply
Download message
Move To:
6 replies: 1 2 3 4 5 6
4 followups: 1 2 3 4

Private message: yes  no

Notes:

Notification:


Date: Wed, 22 Jan 2014 16:53:13 GMT
From: sav2@le.ac.uk
To: xmmhelp@sciops.esa.int
CC: sav2@le.ac.uk
Subject: GTI extensions
Full_Name: Simon Vaughan
Submission from: (NULL) (143.210.38.130)


Dear Helpdesk,

I would like to understand the names and contents of the GTI extensions included
in the FITS event lists produced by e.g. epchain, evselect, etc.

I have read the related discussion (EPICpn/50617) but this did not answer my
question. 

For example, a pn event list from epchain will contain a STDGTI04 extension if
in SW mode (so only CCDNR=4 is used). If I then apply some time filtering in
evselect and produce a filtered event list this contains two GTI extensions:
STDGTI04 and GTI00003. I don't understand why the new one ends in 3 and the old
one in 4 (unless the numbering system for CCDNR changes from 1-12 to 0-11 during
processing!). For the MOS I find STDGTI0x and then GTI00x03 etc.

What is the system for these extension names? 

I would like to know because I use my own software to process filtered event
files, and I need to understand which extensions contain the important GTI
information.

Next question: how are the STDGTI and GTI000 extensions related? 

From the files I have inspected it appears that the GTI000 table includes the
STDGTI intervals as a subset. So the GTI000 extension alone is sufficient to
recover all the good times contained in the event file. Or, applying both STDGTI
and GTI000 is the same as applying GTI000. Is this the case? It is not clear to
me how/if the GTI tables get merged.

Thanks very much for your help.

Simon


Reply 1

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@le.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Thu Jan 23 08:41:52 2014
Dear Simon,

I am in contact with the SAS team discussing your question and will
come back to you as soon as I have any news from them.

Cheers,
   Matthias

--
   Dr Matthias Ehle 
   XMM-Newton Science Operations Centre
   Community Support and Scientific Planning Team


Reply 2

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@le.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Mon Jan 27 09:22:49 2014
Dear Simon,

find herewith information from the SAS developer:

The user is right about the CCDNR numbering of the GTI extensions.
I do not know the reason for this numbering. It has been always this way.

About the question on the difference between STD and GTI extension:
- The STD extension are written by low-level tasks, mainly "epframes" and
"emframes"
- The GTI extension are written by any time filtered expression, mainly
evselect.

In principle, STD and GTI extensions are not related. "evselect" sorts the lists
of GTIs, that's why STD is a subset of the GTI list. But this doesn't mean that
any other task that could create a GTI extension will do the same. 
Furthermore, the GTI extension are not mandatory.

We recommend the user to use both extensions (STD and GTI) to be sure that you 
are using all the good intervals. 

Regards,
   Matthias


Followup 1

Compose reply
Download message
Date: Mon, 27 Jan 2014 11:49:03 +0000
From: Simon Vaughan <sav2@leicester.ac.uk>
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: GTI extensions (PR#73527)
Dear Matthias,

Many thanks for the information.

I would still like further clarification of the naming system.

The standard GTI extensions produced by the processing chains/procs are

For EPIC pn: STDGTInn with nn=01-12 with 04 being the aimpoint chip
For EPIC MOS: STDGTInn with nn=01-07 with 01 being the central chip

But if I apply time filtering of the event list, and keep events from 
only the central/aimpoint CCD, the output of evselect contains 
additional GTI extensions

For EPIC pn I have some files with GTI00405 and some with GTI00005 and 
others with GTI00305.

For EPIC MOS1 and MOS2 I have some with GTI00104 and others with GTI00105.

Can you please explain the naming system for this additional GTI 
extensions. How do I know which extension name to look for?

Cheers,

Simon



Reply 3

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@leicester.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Mon Jan 27 14:07:32 2014
Dear Simon,

we hope the following answers your remaining questions:

1- GTI naming convention: GTIXYYZZ
- YY means the CCD, starting at 0:
  PN: 0-11
  MOS: 0-6

 -ZZ is a sequence number in the Data Subspace corresponding to a TIME filter
expression. If you have filtered your event list using more than one TIME
expression, you will get consecutive GTI extensions (YY02, YY03, etc...)
The event list will always have the STD extension, therefore, the GTI extension
will always start at 02.

2.- Every time you filter your event list using a TIME expression, you will
always get a GTI for each of the CCDs with an increasing ZZ sequence. Therefore,
even if you spatially filter on a single CCD, you will get GTIs for all of the
CCDs. 

Cheers,
  Matthias


Followup 2

Compose reply
Download message
Date: Mon, 27 Jan 2014 16:14:38 +0000
From: Simon Vaughan <sav2@leicester.ac.uk>
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: GTI extensions (PR#73527)
Hi Matthia,

Yes, that has mostly answered by query. But twos question remain for me.

1. Can there be multiple GTI0nnxx extensions in the same filtered event 
list file? In the event lists I have examined I have two GTI extensions 
(per chip) e.g. STDGTI04 and GTI00004 (for pn). There is no GTI00002 or 
GTI00003. Are these always removed, leaving only the GTI0nnxx extensions 
with the largest xx value?

2. If there are multiple GTI0nnxx extensions, how are they related? Does 
GTI0nn04 include the intervals listed in GTI0nn02 as a subset? Or does 
one need to merge all GTI0nnxx GTI tables to find the true GTIs in the 
event list?

Cheers,

Simon



Reply 4

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@leicester.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Tue Jan 28 11:15:33 2014
Dear Simon,

I further discussed your question with our SAS expert; find our reply below:

> 1. Can there be multiple GTI0nnxx extensions in the same filtered event
> list file? 

Yes.

> In the event lists I have examined I have two GTI extensions
> (per chip) e.g. STDGTI04 and GTI00004 (for pn). There is no GTI00002 or
> GTI00003. 

Could you, please, send us the event list? We do not understand yet how it can 
have a GTI00004 without having the corresponding GTI00002. And also, if
possible, 
could you send us the full SAS command sequence used to produce that event
file?

As I said before, the last two digits are simply sequential numbers (blocks)
that correspond to a filter expression containing a TIME (one or several)
intervals.

> Are these always removed, leaving only the GTI0nnxx extensions
> with the largest xx value?

There isn't any SAS task that removes GTIs extensions.

> 2. If there are multiple GTI0nnxx extensions, how are they related?

There isn't any relation between GTI extension.


> Does GTI0nn04 include the intervals listed in GTI0nn02 as a subset? 

No.

> Or does
> one need to merge all GTI0nnxx GTI tables to find the true GTIs in the
> event list?

To find the full range of used GTIs you need to consider all the GTI and STDGTI
extensions.

Regards,
  Matthias


Followup 3

Compose reply
Download message
Date: Tue, 28 Jan 2014 12:34:14 +0000
From: Simon Vaughan <sav2@leicester.ac.uk>
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: GTI extensions (PR#73527)
This is a multi-part message in MIME format.
--------------030804060107010709020902
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Dear Matthias,

Many of my filtered event files that have GTI00004 do not have GTI00003 
or GTI00002. Likewise I have event files with GTI00004 but no GTI00004 
or earlier. I thought this was standard. I also note that in a previous 
Helpdesk discussion (EPICpn/50624) the user had GTI00003 without GTI00002.

For example, some imaging mode pn data from rev1721 processed with 
epchain. I then apply time filtering to the event list as follows:

evselect table=P0606320101PNS003PIEVLI0000.FIT:EVENTS 
filteredset=filtered2.ds expression='((TIME) IN 
[357735964.314:357775195.915])&&(PATTERN<=4)&&(FLAG==0)&&(TIME
in 
gti(gti.ds))'

The output event list file (filtered2.ds) includes only these GTI extensions

HDU 7   STDGTI04           BinTable     2 cols x 8 rows
HDU 9   GTI00004           BinTable     2 cols x 7 rows

As you see there is no GTI00002 or 03.

The event list is rather large (~100M) so I have placed it (before and 
after filtering) here:www.star.le.ac.uk/sav2/temp/ 
<http://www.star.le.ac.uk/sav2/temp/>

If I change the selection expression I can get GTI00002 or GTI0003 or 
GTI0004 but never more than one GTI000xx.

Also, if the system is
> 1- GTI naming convention: GTIXYYZZ
> - YY means the CCD, starting at 0:
>    PN: 0-11
>    MOS: 0-6

Shouldn't the GTIs be called GTI003xx (if the standard GTI is STDGTI04), 
not GTI000xx as I find?

Many thanks,

Simon

--------------030804060107010709020902
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Dear Matthias,<br>
    <br>
    Many of my filtered event files that have GTI00004 do not have
    GTI00003 or GTI00002. Likewise I have event files with GTI00004 but
    no GTI00004 or earlier. I thought this was standard. I also note
    that in a previous Helpdesk discussion (EPICpn/50624) the user had
    GTI00003 without GTI00002.<br>
    <br>
    For example, some imaging mode pn data from rev1721 processed with
    epchain. I then apply time filtering to the event list as follows:<br>
    <br>
    evselect table=P0606320101PNS003PIEVLI0000.FIT:EVENTS
    filteredset=filtered2.ds expression='((TIME) IN
    [357735964.314:357775195.915])&amp;&amp;(PATTERN&lt;=4)&amp;&amp;(FLAG==0)&amp;&amp;(TIME
    in gti(gti.ds))'<br>
    <br>
    The output event list file (filtered2.ds) includes only these GTI
    extensions<br>
    <br>
    HDU 7&nbsp;&nbsp;
STDGTI04&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
BinTable&nbsp;&nbsp;&nbsp;&nbsp; 2 cols x 8
rows&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
    <br>
    HDU 9&nbsp;&nbsp;
GTI00004&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
BinTable&nbsp;&nbsp;&nbsp;&nbsp; 2 cols x 7
rows&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<br>
    <br>
    As you see there is no GTI00002 or 03.<br>
    <br>
    The event list is rather large (~100M) so I have placed it (before
    and after filtering) here:<a
      href="http://www.star.le.ac.uk/sav2/temp/">
      www.star.le.ac.uk/sav2/temp/</a><br>
    <br>
    If I change the selection expression I can get GTI00002 or GTI0003
    or GTI0004 but never more than one GTI000xx.<br>
    <br>
    Also, if the system is <br>
    <blockquote type="cite">
      <pre wrap="">1- GTI naming convention: GTIXYYZZ
- YY means the CCD, starting at 0:
  PN: 0-11
  MOS: 0-6</pre>
    </blockquote>
    <br>
    Shouldn't the GTIs be called GTI003xx (if the standard GTI is
    STDGTI04), not GTI000xx as I find?<br>
    <br>
    Many thanks,<br>
    <br>
    Simon<br>
  </body>
</html>

--------------030804060107010709020902--



Reply 5

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@leicester.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Wed Jan 29 10:35:46 2014
Dear Simon,

the SAS expert provided me with further details:

you wrote:
> Many of my filtered event files that have GTI00004 do not have GTI00003
> or GTI00002. Likewise I have event files with GTI00004 but no GTI00004
> or earlier. I thought this was standard. I also note that in a previous
> Helpdesk discussion (EPICpn/50624) the user had GTI00003 without GTI00002.
>
> For example, some imaging mode pn data from rev1721 processed with
> epchain. I then apply time filtering to the event list as follows:
>
> evselect table=P0606320101PNS003PIEVLI0000.FIT:EVENTS
> filteredset=filtered2.ds expression='((TIME) IN
> [357735964.314:357775195.915])&&(PATTERN<=4)&&(FLAG==0)&&(TIME
> in
> gti(gti.ds))'
>
> The output event list file (filtered2.ds) includes only these GTI
extensions
>
> HDU 7   STDGTI04           BinTable     2 cols x 8 rows
> HDU 9   GTI00004           BinTable     2 cols x 7 rows
>
> As you see there is no GTI00002 or 03.
>
> The event list is rather large (~100M) so I have placed it (before and
> after filtering) here:www.star.le.ac.uk/sav2/temp/
> <http://www.star.le.ac.uk/sav2/temp/>


The sequential last two digits in the GTI extensions depends also on the 
expression filter complexity.
In the example you sent to us, the filter expression has three different
components:
1.- PATTERN
2.- FLAG
3.- TIME

Once selectlilb library reads the expression, it is transformed into a Data 
Subspace model and it is handled by dsslib library. This library organize the 
different components in the expression and assign a sequential number to each 
of them.

This sequential number fixes the order in the data subspace expression.  If you

have a look to the EVENTS header you will see the different: DSTYPX, DSVALX and

DSREFX. That's why in this case you do not have a GTI00002 or GTI00003.

If you just filter with only a TIME component you will see that you have the 
GTI00002 extension + data subspace expression.


> If I change the selection expression I can get GTI00002 or GTI0003 or
> GTI0004 but never more than one GTI000xx.

To get more than one GTI extension you have to use two TIME components in two 
different filter expressions, i.e. call eveselect twice.  For example:

evselect table=P0606320101PNS003PIEVLI0000.FIT withfilteredset=yes
filteredset=filtered.ds expression="((TIME) IN [357735964.314:357775195.915])"

evselect table=filtered.ds withfilteredset=yes filteredset=filtered2.ds
expression="((TIME) IN [357735966.314:357775192.915])"

Here you will have GTI00002 and GTI00003

> Also, if the system is
> > 1- GTI naming convention: GTIXYYZZ
> > - YY means the CCD, starting at 0:
> >    PN: 0-11
> >    MOS: 0-6
>
> Shouldn't the GTIs be called GTI003xx (if the standard GTI is STDGTI04),
> not GTI000xx as I find?


In timing and small window modes, where you only have 1 CCD, the GTI extensions
set the CCD number to 0. 

Regards,
  Matthias


Followup 4

Compose reply
Download message
Date: Wed, 29 Jan 2014 10:51:21 +0000
From: Simon Vaughan <sav2@leicester.ac.uk>
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: GTI extensions (PR#73527)
Hi Matthias,

Many thanks, again, for the email. Wow! The system is rather 
complicated. Is this documented anywhere?

I would like to check one more thing. If I run EVSELECT once on a 
processed event list I will get STDGTI04 and one additional GTI 
extension, e.g. GTI00004. In my experience these seem related. I find 
that the latter GTI list (GTI00004) contains all the information of the 
STDGTI plus additional filtering. What I mean is that there are no good 
times in GTI00004 that are not good in the STDGTI. So in order to know 
the good times for an observation, it would seem that GTI00004 is 
sufficient, merging with STDGTI makes no difference.

Is this true? Is the GTI information in STDGTInn carried forward into 
GTI0nnyy by EVSELECT?

In a previous email you said "In principle, STD and GTI extensions are 
not related." But when I filter/extract real data using EVSELECT I find 
that the GTI0nnyy includes all the good time interval information, 
including any gaps listed in STDGTInn.

I realise that other software (CIAO, FTOOLS) may not follow the same 
rules, but I am specifically interested in the action of EVSELECT.

Cheers,

Simon



Reply 6

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: sav2@leicester.ac.uk
Subject: Re: GTI extensions (PR#73527)
Date: Wed Jan 29 16:11:12 2014
Hi Simon,

the SAS expert provided some more info, which hopefully now answers all
remaining doubts:

> Is this true? Is the GTI information in STDGTInn carried forward into 
> GTI0nnyy by EVSELECT?

As you said before, things can be really complicated. The only way to be 
absolutely sure that you are treating the GTIs correctly is to do an AND on 
all of the GTI and STD extensions for a given CCD.

> In a previous email you said "In principle, STD and GTI extensions are 
> not related." But when I filter/extract real data using EVSELECT I find 
> that the GTI0nnyy includes all the good time interval information, 
> including any gaps listed in STDGTInn.

Again, the GTI and STD extensions are not related. In some cases, depending 
on your filtering expression you can have some GTIs intervals contained in 
different extensions. 

Regards,
  Matthias

Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page