Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing Data/34327
Full headers

From: mhvk@astro.utoronto.ca
Subject: UniidentifiedTimeGaps in 0412600201
Compose reply
Download message
Move To:
4 replies: 1 2 3 4
4 followups: 1 2 3 4

Private message: yes  no

Notes:

Notification:


Date: Tue, 11 Dec 2007 21:31:35 GMT
From: mhvk@astro.utoronto.ca
To: xmmhelp@sciops.esa.int
Cc: mhvk@astro.utoronto.ca
Subject: UniidentifiedTimeGaps in 0412600201
Full_Name: Marten van Kerkwijk
Submission from: (NULL) (128.100.88.12)


Dear Madam, Sir,

while doing a timing analysis of observation an isolated neutron star, an error
message appeared for observation id 0412600201:

** epframes: warning (UnidentifiedTimeGaps), Time intervals were found whose
duration appear not to be integer multiples of the nominal frame integration
time of 0.005672s for the current mode. This hints at either an undetected
random jump of the FTCOARSE time counter or another unknown anomaly with the
auxiliary data from quadrant 1. If a detailed timing analysis is to be carried
out on the event data please note that undetected jumps are very likely to lead
to artifacts in timing data products such as pulse profile curves. Please
contact the SOC for advice.

Since I'm doing timing analysis, obviously I'm worried...

Using SAS_VERBOSITY -5, I got a bit more detail

epframes:- Statistics on auxiliary data from
`././ODF/1330_0412600201_PNU00200AUX.FIT:PNAUX1' for CCD/Quadrant: 0/1 [4] :
epframes:- 
    spurious 32767-frames: 82
    late-reset frames    : 0
    missing frames       : 50657
    normal resets        : 2 [32399/48000(0.02267), 32399/48428(0.02833)]
    premature increments : 1 [15140/48828]
    negative jumps       : 0
    positive jumps       : 2 [24261/48587(1.017-1), 6180/48134(1.034-1)]
    wrong quadrant/CCD   : 8
    unverifiable deltas  : 0
    unidentified deltas  : 30 [3308/39947(-0.9188), 3636/6565(-0.5218),
3785/39084(-0.6693), 4460/41021(-1.049), 7708/17687(-0.9075),
7955/7384(-0.1815), 10906/5084(-1.276), 13500/45891(-1.628),
14836/46310(-0.9188), 19363/48656(-1.01), 22090/25392(-0.8734),
22341/24169(-1.418), 24149/12581(-0.2552), 27724/4327(-0.6296),
30052/394(-0.7203), 31538/38289(-0.7487), 842/39329(-0.6522),
2250/16682(-0.7486), 3513/20152(-0.6693), 7754/895(-0.4481),
12019/1191(-0.8848), 12850/26252(-0.3233), 12969/36178(-0.2269),
14018/26879(-0.2269), 23234/38033(-1.548), 27168/6102(-1.356), 27949/53(-1.605),
30089/8324(-0.777), 30826/4974(-1.032), 1060/22050(-0.7884)]

    minimum delta        : -1.628s
    maximum delta        : 39.25s

From a google, the problem seems somewhat similar to that of 
http://xmm.esac.esa.int/xmmhelp/EPICpn?id=27533;page=19;user=guest

Any ideas?

Thanks,

Marten

p.s.  I found this only while revising a paper on the timing solution.  Without
this observation, our solution becomes better, which makes me even more worried
something is wrong.


Followup 1

Compose reply
Download message
To: xmmhelp@sciops.esa.int
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Cc: dlk@space.mit.edu
From: Marten van Kerkwijk <mhvk@astro.utoronto.ca>
Date: Wed, 12 Dec 2007 00:55:57 -0500
Dear Sir, Madam,

following on from my previous message about epframes warning about
UnidentifiedTimeGaps in ObsID 0412600201, I found from a closer look
at 1330_0412600201_PNU00200AUX.FIT that all problems stem from short
sets of events being present twice (in both the AUX and IME files).  I
give a table below (where all frames with frame1<frame<=frame2 have
times and events that are duplicates of previous frames).

I tried remaking the AUX and IME files by removing all duplicate
frames, and subsequently renumbering FRAME, CYCLE to be consecutive
again.  This indeed led to the disappearance of the
UnidentifiedTimeGaps warnings.  Is this a sensible way to proceed?
Also, is my impression correct that for timing analyses these
duplications should have no effect? (As they'll only cause some events
to be counted twice, but do not actually indicate wrong time
information.)

Finally, earlier I had omitted to give the various other warnings that
epframes gave.  I now include these below.

With all best wishes,

Marten

-- 
Prof. M. H. van Kerkwijk
Dept. of Astronomy & Astroph., 50 St George St., Toronto, ON, M5S 3H4,
Canada
McLennan Labs, room 1404D, tel: +1(416)9467288, fax: +1(416)9467287
mhvk@astro.utoronto.ca, http://www.astro.utoronto.ca/~mhvk

-=-=-=-=- duplicated event times -=-=-=-=-=-

All frames with frame1<frame<=frame2 are duplicates of the preceding ones.

  Table : testdup 

 Sequence ftcoa ftfin dt     frame1   frame2  
 -------- ----- ----- ------ -------- --------
        1  3308 39947 -0.918   106460   106492
        2  3636  6565 -0.521   116366   116382
        3  3785 39084 -0.669   121112   121137
        4  4460 41021 -1.049   141769   141802
        5  7708 17687 -0.907   240469   240506
        6  7955  7384 -0.181   249938   249949
        7 10906  5084 -1.276   355876   355911
        8 13500 45891 -1.628   437325   437363
        9 14836 46310 -0.918   479613   479649
       10 19363 48656 -1.010   629783   629812
       11 22090 25392 -0.873   718406   718436
       12 22341 24169 -1.418   725831   725872
       13 24149 12581 -0.255   783393   783402
       14 27725  4327 -0.629   888680   888700
       15 30053   394 -0.720   963348   963379
       16 31539 38289 -0.748  1009618  1009635
       17   842 39329 -0.652  1060402  1060431
       18  2250 16682 -0.748  1114846  1114875
       19  3513 20152 -0.669  1156631  1156669
       20  7755   895 -0.448  1297203  1297217
       21 12020  1191 -0.884  1445640  1445673
       22 12851 26252 -0.323  1475965  1475979
       23 12970 36178 -0.226  1479509  1479520
       24 14019 26879 -0.226  1513123  1513130
       25 19175 29642      *  1655344  1655349
       26 23235 38033 -1.548  1768956  1768998
       27 27169  6102 -1.356  1881161  1881202
       28 27950    53 -1.605  1903388  1903441
       29 30090  8324 -0.777  1964831  1964862
       30 30827  4974 -1.032  1988160  1988192
       31  1060 22050 -0.788  2064733  2064757
 -------- ----- ----- ------ -------- --------

Note: entry 25 was not found by epframes, but looking at all cases for
which FTCOARSE+FTFINE*20.48e-6 jumped down, I identified it myself as
a five-frame duplication.  For this particular one (I did not check
all sets), frame 1655345 seems to have substantially more events than
1655340 (the latter has 8, of which the last 5 are identical to the
last 5 of 1655345, which has 215 events).


-=-=-=- Other warnings given by epframes -=-=-=-=-

././ODF/1330_0412600201_PNU00204IME.FIT
** epframes: warning (UnidentifiedTimeGaps), Time intervals were found whose
duration appear not to be integer multiples of the nominal frame integration
time of 0.005672s for the current mode. This hints at either an undetected
random jump of the FTCOARSE time counter or another unknown anomaly with the
auxiliary data from quadrant 1. If a detailed timing analysis is to be carried
out on the event data please note that undetected jumps are very likely to lead
to artifacts in timing data products such as pulse profile curves. Please
contact the SOC for advice.
** epframes: warning (InvalidObtValue), OBT vector element #1654611 is invalid
(-100)
** epframes: warning (InvalidObtValue), OBT vector element #1654612 is invalid
(-100)
** epframes: warning (InvalidObtValue), OBT vector element #1654613 is invalid
(-100)
** epframes: warning (illegalODF),  The PMH file seems to end prematurely:
DATE-END > TIME(PMH)_max: [2007-03-15T15:21:03] [2007-03-15T15:17:21]   The
ODF is invalid, please contact the originator of this data.
** epframes: warning (notHKconstant),  Range of filter wheel position sensor
values F1122 [ 54.736,241.610] exceeds limits [111.720,116.748] for average
value 113.437
** epframes: warning (illegalODF),  The PAH file seems to end prematurely:
DATE-END > TIME(PAH)_max: [2007-03-15T15:21:03] [2007-03-15T15:17:17]   The
ODF is invalid and cannot be processed properly. Please contact the originator
of this data.
** epframes: warning (BPTcode),  BadPixelTab

Message of length 7327 truncated


Reply 1

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: mhvk@astro.utoronto.ca
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Date: Wed Dec 12 14:03:43 2007
Dear Marten,

To access the problem we would need more detailed information 
from you.

Can you, please, provide us with
- all obs ids you are using
- info on what exact timing analysis steps you are performing
- a clarification of what exactly you mean when saying 
  "without this observation, our solution becomes better".

Kind regards,
  Matthias

--
   Dr Matthias Ehle 
   XMM-Newton User Support Group
   European Space Astronomy Centre


Followup 2

Compose reply
Download message
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Cc: dlk@space.mit.edu
From: Marten van Kerkwijk <mhvk@astro.utoronto.ca>
Date: Wed, 12 Dec 2007 10:42:02 -0500
Dear Matthias,

Before anything else, I thought I should note that 0412600201 seems
odd for MOS as well: for both M1 and M2, there are many warnings
similar to:

** emframes: warning (frames17),  non increasing time after frame number  6938

This seems to suggest there might have been a more general telemetry
problem, where packets were received multiple times or so (I have not
yet investigated, however, whether these warnings are again due to
duplicated data and whether they occur at the same time; I did check
that they do not occur for any other observation we looked at).

> To access the problem we would need more detailed information 
> from you.
> 
> Can you, please, provide us with
> - all obs ids you are using

We use all observations of RX J1856.5-3754, and all exposures of these
suitable for timing its 7 s pulsations (all but full-frame M1,M2 in
0201590101). 

 0106260101M1S002
 0106260101M2S003
 0106260101PNS001
 0165971601M1S001
 0165971601M2S002
 0165971601PNS003
 0165971701M1S001
 0165971701M2S002
 0165971901M1S001
 0165971901M2S002
 0165971901PNS003
 0165972001M1S001
 0165972001M2S002
 0165972001PNS003
 0165972101M1S001
 0165972101M2S002
 0165972101PNS003
 0201590101PNS003
 0213080101M1U002
 0213080101M2U002
 0412600101M1S001
 0412600101M2S002
 0412600101PNS003
 0412600201M1S001
 0412600201M2S002
 0412600201PNU002
 0412600301M1S001
 0412600301M1U002
 0412600301M2S002
 0412600301M2U002
 0412600301PNS003
 0412600301PNU002
 0415180101M1S002
 0415180101M2S003
 0415180101PNS001
 

> - info on what exact timing analysis steps you are performing

In essence, "e[mp]chain + evselect-for-source + barycen" with some
special settings to maximize the number of counts.  The resulting
event list is then subjected to the usual power spectrum analysis and
arrival times are derived.  Details can be found in the Letter which
we submitted earlier (and the revision of which triggered all this);
see

http://www.astro.utoronto.ca/~mhvk/rxj1856timing.pdf

> - a clarification of what exactly you mean when saying 
>   "without this observation, our solution becomes better".

In our solution, we fit both arrival times and frequencies.  For the
frequencies, we find that obsid 0412600201 has the most deviant one.
A fit of f,fdot to our 11 frequencies gives chi2=14.9/9 including it,
and chi2=6.7/8 without it.  Note, however, that the arrival time does
not stand out, and also the precise solution does not depend on
whether or not we include 0412600201, so this could simply be a
statistical fluke.

All best wishes,

Marten

p.s.  In an earlier analysis, of the isolated neutron star RX
J0720.4-3125, we also had found duplicated frames.  This was for 
0622_0158360201_PNU00200AUX.FIT, where frames 963685--963719 were
duplicates of what went before.  See Kaplan & Van Kerkwijk, 2005,
http://adsabs.harvard.edu/abs/2005ApJ...628L..45K 



Followup 3

Compose reply
Download message
To: Matthias Ehle <xmmhelp@sciops.esa.int>
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
From: Marten van Kerkwijk <mhvk@astro.utoronto.ca>
Date: Wed, 12 Dec 2007 14:27:32 -0500
Dear Matthias,

following up on my earlier comment that the MOS also seem to show
unusual time jumps: I just checked and the frames17 warnings also
correspond to series of events occuring twice.

For instance, for M1, CCDID 2, the times and events associated with
frames 4283 and 4299 are identical.

I could not find an obvious correspondence in time between these
occurences for different instruments/CCDs.

All the best,

Marten



Reply 2

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: mhvk@astro.utoronto.ca
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Date: Thu Dec 13 09:14:39 2007
Dear Marten,

Our EPIC expert is already working on the data. We will let you
have results of his analysis as soon as possible.

Greetings,
  Matthias


Reply 3

Resend
From: Nora Loiseau <xmmhelp@sciops.esa.int>
To: mhvk@astro.utoronto.ca
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Date: Fri Dec 14 11:04:14 2007
Dear Dr. van Kerkwijk

here is the reply by EPIC calibration scientists about this problem.

============
"epframes" cannot do anything against it.
As already mentioned in the analysis, "emframes" is affected as well,
pointing towards a common origin of the problem, e.g. SAS infrastructure
(OAL) or data (time correlation files in ODF).

SAS_JUMP_TOLERANCE=22 does not change anything, same jumps in the data.
TCS instead of TCX does not change anything, same jumps in the data.
SAS_JUMP_TOLERANCE=3 does not change significantly the output, same
number of jumps.
============

Their recommendation would then be not to include this observation in 
your timing analysis because of the delay that would imply to wait
for ODF reprocessing or OAL updates. 

Thanking you for reporting this problem,

Sincerely,

Nora
----
Dr. Nora Loiseau
XMM-Newton User Support Group


Followup 4

Compose reply
Download message
To: Nora Loiseau <xmmhelp@sciops.esa.int>
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Cc: dlk@space.mit.edu
From: Marten van Kerkwijk <mhvk@astro.utoronto.ca>
Date: Fri, 14 Dec 2007 11:34:53 -0500
Dear Nora, Matthias,

> here is the reply by EPIC calibration scientists about this problem.
> 
> ============
> "epframes" cannot do anything against it.
> As already mentioned in the analysis, "emframes" is affected as well,
> pointing towards a common origin of the problem, e.g. SAS infrastructure
> (OAL) or data (time correlation files in ODF).
> 
> SAS_JUMP_TOLERANCE=22 does not change anything, same jumps in the data.
> TCS instead of TCX does not change anything, same jumps in the data.
> SAS_JUMP_TOLERANCE=3 does not change significantly the output, same
> number of jumps.
> ============
> 
> Their recommendation would then be not to include this observation in 
> your timing analysis because of the delay that would imply to wait
> for ODF reprocessing or OAL updates. 

Thanks for the quick turnaround.  From my inspections of the ODF, it
seems pretty clear the ODF is the underlying problem, so I guess one
would indeed need to wait for ODF processing.  Anyway, since the
observation is actually consistent with our solution, I guess I'll
include it, having removed the duplicate frames, but mention the
problems. 

Finally, a question: Is there any way to know whether an ODF has been
updated without actually downloading the whole set (i.e., can one
access the ODF "version number" somewhere via, say, XSA?).

With all best wishes,

Marten


> Thanking you for reporting this problem,
> 
> Sincerely,
> 
> Nora
> ----
> Dr. Nora Loiseau
> XMM-Newton User Support Group
> 
> ================================================================================================
> This message and any attachments are intended for the use of the addressee
or addressees only. The
> unauthorised disclosure, use, dissemination or copying (either in whole or
in part) of its content
> is prohibited. If you received this message in error, please delete it from
your system and notify
> the sender. E-mails can be altered and their integrity cannot be
guaranteed. ESA shall not be liable
> for any e-mail if modified.
> =================================================================================================
> 



Reply 4

Resend
From: Matthias Ehle <xmmhelp@sciops.esa.int>
To: mhvk@astro.utoronto.ca
Subject: Re: UniidentifiedTimeGaps in 0412600201 (PR#34327)
Date: Mon Dec 17 13:08:28 2007
Dear Marten,

you wrote:
> Finally, a question: Is there any way to know whether an ODF has been
> updated without actually downloading the whole set (i.e., can one
> access the ODF "version number" somewhere via, say, XSA?).

In XSA you can check which version was used to generate the ODF:
Select 'Configuration Info' in the top right window of the XSA
interface after having executed a query for a specific observation.

The version of the Observation Data System, with which the ODF was 
produced, is displayed. This version number should increase if a
modified s/w was used to re-process the ODF.

Otherwise you in fact need to download the ODF and look at the
so-called MANIFEST file which contains info on the ODF version
and creation date. The version is also given in the *SUM.ASC ODF 
file - look for "ODF_VERSION".


Greetings,
   Matthias 


Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page