Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing esas/43030
Full headers

From: briggs@astro.phys.ethz.ch
Subject: ESAS mos-spectra: run time
Compose reply
Download message
Move To:
6 replies: 1 2 3 4 5 6
6 followups: 1 2 3 4 5 6

Private message: yes  no

Notes:

Notification:


Date: Wed, 3 Feb 2010 09:03:49 GMT
From: briggs@astro.phys.ethz.ch
To: xmmhelp@sciops.esa.int
Cc: briggs@astro.phys.ethz.ch
Subject: ESAS mos-spectra: run time
Full_Name: Kevin Briggs
Submission from: (NULL) (90.26.239.229)


Dear Helpdesk

ESAS mos-spectra has been running overnight on my Mac (OSX 10.4.11 on a 2 GHz
PowerPC G5) and has only reached ccd 2. The results look ok so far and the
output log file reports no errors, but I was wondering if that is a normal
length of time required to make products for the full field. If so, it would be
helpful to note that in the task documentation.

The command used is:

mos-spectra prefix="1U002" mask=1 elow=300 ehigh=1000 ccd1=1 ccd2=1 ccd3=1
ccd4=1 ccd5=1 ccd6=0 ccd7=1 caldb="/usr/local/sas/ccf/esas/mos" region=none
>&
mos-spectra_m1.log

I include the full log file produced thus far below. My impression is that none
of the SAS tasks with reported start and end times seem to take a long time but
something takes a long time in between. 

Here is a possible relevant snippet from the log file, where one backscale ends
at 23:38 while the next doesn't start until 06:40 (a similar thing happened
between 21:45 and 23:02): 

** backscale::arfgen::attcalc: warning (outOfImageRange),  305  events ( of 
1102500  ) out of range of image
backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0] ended: 
  2010-02-02T23:38:40.000
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX01:RAWX and temp_badcol_arfgen.del:BADPIX01:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX02:RAWX and temp_badcol_arfgen.del:BADPIX02:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX03:RAWX and temp_badcol_arfgen.del:BADPIX03:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX04:RAWX and temp_badcol_arfgen.del:BADPIX04:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX05:RAWX and temp_badcol_arfgen.del:BADPIX05:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- WCS info for the columns
temp_badcol_arfgen.del:BADPIX07:RAWX and temp_badcol_arfgen.del:BADPIX07:RAWX is
either non-existent or incomplete. Assuming a 1:1 mapping
backscale::arfgen:- Writing energy data into dataset .. 
backscale::arfgen:- Calculating Area curve ...
backscale::arfgen:- Calculating Area curve ...2.5 2400 11997.5
backscale::arfgen:- Executing (routine): attcalc eventset=rawpixellist.ds:EVENTS
fixedra=83.437625 fixeddec=-5.692861 fixedposangle=78.600143 attitudelabel=fixed
nominalra=83.437625 nominaldec=-5.692861 refpointlabel=user atthkset=atthk.dat
withatthkset=no withmedianpnt=yes imagesize=0.36  -w 3 -V 4
backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0]
started:  2010-02-03T06:40:43.000
backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0] ended: 
  2010-02-03T06:40:43.000

I look forward to your reply.
Kind regards
Kevin  

Full log file follows (NOTE task has not yet  completed):

ccd 1

Task: mos-spectra 
Version: 3.0 
Date: 2008-08-20 
Author: S. L. Snowden, based on code from K. D. Kuntz 

prefix: 1U002
fwc_dir: /usr/local/sas/ccf/esas/mos
regfile: none
backmask: 1
elow: 300
ehigh: 1000
det: mos1
ccds: 1 1 1 1 1 0 1 
   
detpref: mos1U002-clean.fits
Revolution Number: 1780
   
mos1U002-bkg_region-sky.fits 
Region file exists.  Proceed with source exclusion. 
maskit: &&region(mos1U002-bkg_region-sky.fits)
ccddef: &&!BOX(-13169,-105,6599,6599,0,DETX,DETY)
ccddefa:  &&((CCDNR == 1)||(CCDNR == 2)||(CCDNR == 3)||(CCDNR ==
4)||(CCDNR ==
5)||(CCDNR == 7))
none not found
Will default to no additional selection
   
Area Selection: 
   
corner event file already exists: mos1U002-corn.fits
   
image file already exists: mos1U002-obj-im.fits
   
evselect table=mos1U002-clean.fits:EVENTS withfilteredset=yes
expression='(PATTERN<=12)&&(FLAG == 0)&&(((DETX,DETY) IN
box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
circle(97,-172,17152)))&&!BOX(-13169,-105,6599,6599,0,DETX,DETY)&&region(mos1U002-bkg_region-sky.fits)'
filtertype=expression imagebinning='imageSize' imagedatatype='Int32'
imageset=mos1U002-obj-im-sp-det.fits squarepixels=yes ignorelegallimits=yes
withxranges=yes withyranges=yes xcolumn='DETX' ximagesize=780 ximagemax=19500
ximagemin=-19499 ycolumn='DETY' yimagesize=780 yimagemax=19500 yimagemin=-19499
updateexposure=yes filterexposure=yes
   
evselect:- Executing (routine): evselect table=mos1U002-clean.fits:EVENTS
filteredset=filtered.fits withfilteredset=yes keepfilteroutput=yes
flagcolumn=EVFLAG flagbit=-1 destruct=yes dssblock=''
expression='(PATTERN<=12)&&(FLAG == 0)&&(((DETX,DETY) IN
box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
box(2743.5,-16051,2579.5,1340,0))|

Message of length 90019 truncated

Reply 1

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Wed Feb  3 13:55:08 2010
Dear Kevin,

could you please send me the Obs ID that you are using? I will try to reproduce
your results here. I have so far checked your problem with the SAS experts and
they tell me that it looks rather as if your computer had few CPU available over
night, especially the fact that there is so much time between the end of one
task and the beginning of the next one. Could it be that a backup task or
similar was running at the same time?

Best wishes,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> Full_Name: Kevin Briggs
> Submission from: (NULL) (90.26.239.229)
> 
> 
> Dear Helpdesk
> 
> ESAS mos-spectra has been running overnight on my Mac (OSX 10.4.11 on a 2
GHz
> PowerPC G5) and has only reached ccd 2. The results look ok so far and the
> output log file reports no errors, but I was wondering if that is a normal
> length of time required to make products for the full field. If so, it
would
be
> helpful to note that in the task documentation.
> 
> The command used is:
> 
> mos-spectra prefix="1U002" mask=1 elow=300 ehigh=1000 ccd1=1 ccd2=1 ccd3=1
> ccd4=1 ccd5=1 ccd6=0 ccd7=1 caldb="/usr/local/sas/ccf/esas/mos" region=none
>&
> mos-spectra_m1.log
> 
> I include the full log file produced thus far below. My impression is that
none
> of the SAS tasks with reported start and end times seem to take a long time
but
> something takes a long time in between. 
> 
> Here is a possible relevant snippet from the log file, where one backscale
ends
> at 23:38 while the next doesn't start until 06:40 (a similar thing
happened
> between 21:45 and 23:02): 
> 
> ** backscale::arfgen::attcalc: warning (outOfImageRange),  305  events ( of

> 1102500  ) out of range of image
> backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0]
ended: 
>   2010-02-02T23:38:40.000
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX01:RAWX and
temp_badcol_arfgen.del:BADPIX01:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX02:RAWX and
temp_badcol_arfgen.del:BADPIX02:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX03:RAWX and
temp_badcol_arfgen.del:BADPIX03:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX04:RAWX and
temp_badcol_arfgen.del:BADPIX04:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX05:RAWX and
temp_badcol_arfgen.del:BADPIX05:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX07:RAWX and
temp_badcol_arfgen.del:BADPIX07:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- Writing energy data into dataset .. 
> backscale::arfgen:- Calculating Area curve ...
> backscale::arfgen:- Calculating Area curve ...2.5 2400 11997.5
> backscale::arfgen:- Executing (routine): attcalc
eventset=rawpixellist.ds:EVENTS
> fixedra=83.437625 fixeddec=-5.692861 fixedposangle=78.600143
attitudelabel=fixed
> nominalra=83.437625 nominaldec=-5.692861 refpointlabel=user
atthkset=atthk.dat
> withatthkset=no withmedianpnt=yes imagesize=0.36  -w 3 -V 4
> backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0]
> started:  2010-02-03T06:40:43.000
> backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0]
ended: 
>   2010-02-03T06:40:43.000
> 
> I look forward to your reply.
> Kind regards
> Kevin  
> 
> Full log file follows (NOTE task has not yet  completed):
> 
> ccd 1
> 
> Task: mos-spectra 
> Version: 3.0 
> Date: 2008-08-20 
> Author: S. L. Snowden, based on code from K. D. Kuntz 
> 
> prefix: 1U002
> fwc_dir: /usr/local/sas/ccf/esas/mos
> regfile: none
> backmask: 1
> elow: 300
> ehigh: 1000
> det: mos1
> ccds: 1 1 1 1 1 0 1 
>    
> detpref: mos1U002-clean.fits
> Revolution Number: 1780
>    
> mos1U002-bkg_region-sky.fits 
> Region file exists.  Proceed with source exclusion. 
> maskit: &&region(mos1U002-bkg_region-sky.fits)
> ccddef: &&!BOX(-13169,-105,6599,6599,0,DETX,DETY)
> ccddefa:  &&((CCDNR == 1)||(CCDNR == 2)||(CCDNR == 3)||(CCDNR ==
4)||(CCDNR
==
> 5)||(CCDNR == 7))
> none not found
> Will default to no additional selection
>    
> Area Selection: 
>    
> corner event file already exists: mos1U002-corn.fits
>    
> image file alr

Message of length 99880 truncated


Followup 1

Compose reply
Download message
Subject: AW: ESAS mos-spectra: run time (PR#43030)
Date: Wed, 3 Feb 2010 16:18:48 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
Dear Maria

Many thanks for your swift reply. 

The ObsID is: 0605900101 (a proprietary data set)

I will look into the possibility that a backup took up CPU overnight.

The task is still running, however, now processing CCD 5.
Whenever I do a 'ps u', backscale seems to be running, taking 95% 
of available CPU. 

I can now clarify there is no rest time between tasks. I apologise I hadn't
looked 
closely enough at the log file to spot that backscale makes several calls 
to other tasks, which have their start and end times before backscale ends.

Instead, each backscale run on the '#obj.pi' file takes 1+ hours. 

obj 20:34 - 21:34
1obj 23:36 - 06:47 (is ccd1 more complex due to the patch?)
2obj 06:50 - 08:40
3obj 08:57 - 11.36
4obj 12:01 - 13:25
5obj 13:41 - 14:56

Even if the overnight 1obj run was anomalous, the task would still take over 12
hours to complete. 
Which is ok as long as you know in advance (and don't need to run it too many
times). 
But is that a normal run time for this task? I could just have a slow computer,
of course. 

Maybe the complex region expression is a factor?
e.g.
expression='(PATTERN<=12)&&(FLAG == 0)&&(((DETX,DETY) IN
box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
circle(97,-172,17152)))&&BOX(-6435.5,13094,6633,6599,0,DETX,DETY)&&!BOX(-13169,-105,6599,6599,0,DETX,DETY)&&region(mos1U002-bkg_region-sky.fits)'

I'm a little surprised with all those DETX, DETY definitions that the
*region-sky.fits file and not the *region-det.fits file appears here.
There are 181 regions in the region file, BTW.

Thanks for looking into this, I look forward to your reply.
Kind regards
Kevin

-----Urspr.ngliche Nachricht-----
Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
Gesendet: Mi 03.02.2010 14:55
An: Briggs  Kevin Roy
Betreff: Re: ESAS mos-spectra: run time (PR#43030)
 
Dear Kevin,

could you please send me the Obs ID that you are using? I will try to reproduce
your results here. I have so far checked your problem with the SAS experts and
they tell me that it looks rather as if your computer had few CPU available over
night, especially the fact that there is so much time between the end of one
task and the beginning of the next one. Could it be that a backup task or
similar was running at the same time?

Best wishes,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> Full_Name: Kevin Briggs
> Submission from: (NULL) (90.26.239.229)
> 
> 
> Dear Helpdesk
> 
> ESAS mos-spectra has been running overnight on my Mac (OSX 10.4.11 on a 2
GHz
> PowerPC G5) and has only reached ccd 2. The results look ok so far and the
> output log file reports no errors, but I was wondering if that is a normal
> length of time required to make products for the full field. If so, it
would
be
> helpful to note that in the task documentation.
> 
> The command used is:
> 
> mos-spectra prefix="1U002" mask=1 elow=300 ehigh=1000 ccd1=1 ccd2=1 ccd3=1
> ccd4=1 ccd5=1 ccd6=0 ccd7=1 caldb="/usr/local/sas/ccf/esas/mos" region=none
>&
> mos-spectra_m1.log
> 
> I include the full log file produced thus far below. My impression is that
none
> of the SAS tasks with reported start and end times seem to take a long time
but
> something takes a long time in between. 
> 
> Here is a possible relevant snippet from the log file, where one backscale
ends
> at 23:38 while the next doesn't start until 06:40 (a similar thing happened
> between 21:45 and 23:02): 
> 
> ** backscale::arfgen::attcalc: warning (outOfImageRange),  305  events ( of

> 1102500  ) out of range of image
> backscale::arfgen:- attcalc (attcalc-4.32)  [xmmsas_20090615_1803-9.0.0]
ended: 
>   2010-02-02T23:38:40.000
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX01:RAWX and
temp_badcol_arfgen.del:BADPIX01:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX02:RAWX and
temp_badcol_arfgen.del:BADPIX02:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX03:RAWX and
temp_badcol_arfgen.del:BADPIX03:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX04:RAWX and
temp_badcol_arfgen.del:BADPIX04:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX05:RAWX and
temp_badcol_arfgen.del:BADPIX05:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 mapping
> backscale::arfgen:- WCS info for the columns
> temp_badcol_arfgen.del:BADPIX07:RAWX and
temp_badcol_arfgen.del:BADPIX07:RAWX
is
> either non-existent or incomplete. Assuming a 1:1 map

Message of length 100861 truncated


Reply 2

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Thu Feb  4 16:39:35 2010
Dear Kevin,

I am looking in detail at your observation. The times that you report are
definitely not OK. The tasks should run within minutes. I presume that the
problems come already from the fact that 'cheese' is not running properly, as
specified in your next ticket (#43035), so let's first concentrate in the other
problem.

For this, could you please send us the full log that you get when you execute
cheese? And could you please run it with verbosity 5? I have run here cheese and
it does seem to work, though there is an issue with a big hole in the output
image which I will come back to. But first, we should understand what is
happening when you run cheese.

Please also note that ESAS cannot be used for pn, see 
http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.shtml for
more info on this issue.

Looking forward to your reply,

Maria

Maria Diaz Trigo
XMM-Newton SOC



> 
> Dear Maria
> 
> Many thanks for your swift reply. 
> 
> The ObsID is: 0605900101 (a proprietary data set)
> 
> I will look into the possibility that a backup took up CPU overnight.
> 
> The task is still running, however, now processing CCD 5.
> Whenever I do a 'ps u', backscale seems to be running, taking 95% 
> of available CPU. 
> 
> I can now clarify there is no rest time between tasks. I apologise I
hadn't
> looked 
> closely enough at the log file to spot that backscale makes several calls 
> to other tasks, which have their start and end times before backscale
ends.
> 
> Instead, each backscale run on the '#obj.pi' file takes 1+ hours. 
> 
> obj 20:34 - 21:34
> 1obj 23:36 - 06:47 (is ccd1 more complex due to the patch?)
> 2obj 06:50 - 08:40
> 3obj 08:57 - 11.36
> 4obj 12:01 - 13:25
> 5obj 13:41 - 14:56
> 
> Even if the overnight 1obj run was anomalous, the task would still take
over
12
> hours to complete. 
> Which is ok as long as you know in advance (and don't need to run it too
many
> times). 
> But is that a normal run time for this task? I could just have a slow
computer,
> of course. 
> 
> Maybe the complex region expression is a factor?
> e.g.
> expression='(PATTERN<=12)&&(FLAG == 0)&&(((DETX,DETY)
IN
> box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
> box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
> circle(97,-172,17152)))&&BOX(-6435.5,13094,6633,6599,0,DETX,DETY)&&!BOX(-13169,-105,6599,6599,0,DETX,DETY)&&region(mos1U002-bkg_region-sky.fits)'
> 
> I'm a little surprised with all those DETX, DETY definitions that the
> *region-sky.fits file and not the *region-det.fits file appears here.
> There are 181 regions in the region file, BTW.
> 
> Thanks for looking into this, I look forward to your reply.
> Kind regards
> Kevin
> 
> -----Urspr.ngliche Nachricht-----
> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Gesendet: Mi 03.02.2010 14:55
> An: Briggs  Kevin Roy
> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
>  
> Dear Kevin,
> 
> could you please send me the Obs ID that you are using? I will try to
reproduce
> your results here. I have so far checked your problem with the SAS experts
and
> they tell me that it looks rather as if your computer had few CPU available
over
> night, especially the fact that there is so much time between the end of
one
> task and the beginning of the next one. Could it be that a backup task or
> similar was running at the same time?
> 
> Best wishes,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> Full_Name: Kevin Briggs
>> Submission from: (NULL) (90.26.239.229)
>> 
>> 
>> Dear Helpdesk
>> 
>> ESAS mos-spectra has been running overnight on my Mac (OSX 10.4.11 on a
2
GHz
>> PowerPC G5) and has only reached ccd 2. The results look ok so far and
the
>> output log file reports no errors, but I was wondering if that is a
normal
>> length of time required to make products for the full field. If so, it
would
> be
>> helpful to note that in the task documentation.
>> 
>> The command used is:
>> 
>> mos-spectra prefix="1U002" mask=1 elow=300 ehigh=1000 ccd1=1 ccd2=1
ccd3=1
>> ccd4=1 ccd5=1 ccd6=0 ccd7=1 caldb="/usr/local/sas/ccf/esas/mos"
region=none
>>&
>> mos-spectra_m1.log
>> 
>> I include the full log file produced thus far below. My impression is
that
> none
>> of the SAS tasks with reported start and end times seem to take a long
time
> but
>> something takes a long time in between. 
>> 
>> Here is a possible relevant snippet from the log file, where one
backscale
> ends
>> at 23:38 while the next doesn't start until

Message of length 111575 truncated


Followup 2

Compose reply
Download message
Subject: AW: ESAS mos-spectra: run time (PR#43030)
Date: Thu, 4 Feb 2010 19:47:49 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
Dear Maria,

I think 'cheese' ran ok except for not producing the mask at the end.
It produced a reasonable region list, with the exception of the big 
exclusion region, which I edited down to a radius of 300 units (15 arcsec).
(At least I edited the *region-sky.fits file, I forgot to edit the
*region-det.fits 
file, but the mos-spectra output log indicates that mos-spectra only uses the 
*region-sky.fits file.)

I think 181 sources is reasonable for the length of exposure (70 ks good time) 
and field (nr Orion). 

The 'cheese' problem reported occurred in 'make_mask'. 
I wonder if there's a problem generally with the esas fortran procedures on my 
system. If I run make_mask alone I get the same error. 

The most notable part 
of the log file is that the image coordinate ID in the Template file 
(mos1U002-mask-im.fits) is reported as gibberish before the error occurs 
(although the FITS header of that file looks fine) and I wonder if the 
coordinate system is supposed to be reported as "-TAN":

 Template File
 mos1U002-mask-im.fits                                                          

 Template image coordinate system: -TAN
 Template image coordinate ID: ....D  1.D.4
 Template image central coordinates:   83.4376249999999970  -5.6928611111111103
 Template image reference pixel:    4.5091000000000003E+02  
4.5091000000000003E+02
 Template image pixel sizes:   -6.9444444444444501E-04   6.9444444444444501E-04
 Template image NAXES, NAXIS:  2 900 900
 mos1U002-bkg_region-sky.fits                                                   

 6 1 2 3 4
 File exists with output file name
 Clobber the file and continue
 STATUS various keyword writing = 207
 MAKE-MASK POSITION 27
 mos1U002-cheese.fits      

Thanks for the reminder re. pn. I had spotted that. I wanted to see how the 
filtering and source detection part of ESAS worked on the pn data. (These 
parts - pn-filter and cheese - work ok for pn data, right?)

I'm rerunning cheese with -v=5 and will include the log file in my next mail.

Thanks for your efforts
Kevin




-----Urspr.ngliche Nachricht-----
Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
Gesendet: Do 04.02.2010 17:39
An: Briggs  Kevin Roy
Betreff: Re: ESAS mos-spectra: run time (PR#43030)
 
Dear Kevin,

I am looking in detail at your observation. The times that you report are
definitely not OK. The tasks should run within minutes. I presume that the
problems come already from the fact that 'cheese' is not running properly, as
specified in your next ticket (#43035), so let's first concentrate in the other
problem.

For this, could you please send us the full log that you get when you execute
cheese? And could you please run it with verbosity 5? I have run here cheese and
it does seem to work, though there is an issue with a big hole in the output
image which I will come back to. But first, we should understand what is
happening when you run cheese.

Please also note that ESAS cannot be used for pn, see 
http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.shtml for
more info on this issue.

Looking forward to your reply,

Maria

Maria Diaz Trigo
XMM-Newton SOC



================================================================================================
This message and any attachments are intended for the use of the addressee or
addressees only.

The unauthorised disclosure, use, dissemination or copying (either in whole or
in part) of its content is prohibited.

If you received this message in error, please delete it from your system and
notify the sender.

E-mails can be altered and their integrity cannot be guaranteed.

ESA shall not be liable for any e-mail if modified.
=================================================================================================





Followup 3

Compose reply
Download message
Subject: AW: ESAS mos-spectra: run time (PR#43030)
Date: Thu, 4 Feb 2010 23:29:21 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
This is a multi-part message in MIME format.

------_=_NextPart_001_01CAA5E9.C6D964F0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Dear Maria

Please find the output log from cheese (verb=3D5) attached.

Kevin


-----Urspr=FCngliche Nachricht-----
Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
Gesendet: Do 04.02.2010 17:39
An: Briggs  Kevin Roy
Betreff: Re: ESAS mos-spectra: run time (PR#43030)
=20
Dear Kevin,

I am looking in detail at your observation. The times that you report =
are
definitely not OK. The tasks should run within minutes. I presume that =
the
problems come already from the fact that 'cheese' is not running =
properly, as
specified in your next ticket (#43035), so let's first concentrate in =
the other
problem.

For this, could you please send us the full log that you get when you =
execute
cheese? And could you please run it with verbosity 5? I have run here =
cheese and
it does seem to work, though there is an issue with a big hole in the =
output
image which I will come back to. But first, we should understand what is
happening when you run cheese.

Please also note that ESAS cannot be used for pn, see=20
http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.sht=
ml for
more info on this issue.

Looking forward to your reply,

Maria

Maria Diaz Trigo
XMM-Newton SOC



>=20
> Dear Maria
>=20
> Many thanks for your swift reply.=20
>=20
> The ObsID is: 0605900101 (a proprietary data set)
>=20
> I will look into the possibility that a backup took up CPU overnight.
>=20
> The task is still running, however, now processing CCD 5.
> Whenever I do a 'ps u', backscale seems to be running, taking 95%=20
> of available CPU.=20
>=20
> I can now clarify there is no rest time between tasks. I apologise I =
hadn't
> looked=20
> closely enough at the log file to spot that backscale makes several =
calls=20
> to other tasks, which have their start and end times before backscale =
ends.
>=20
> Instead, each backscale run on the '#obj.pi' file takes 1+ hours.=20
>=20
> obj 20:34 - 21:34
> 1obj 23:36 - 06:47 (is ccd1 more complex due to the patch?)
> 2obj 06:50 - 08:40
> 3obj 08:57 - 11.36
> 4obj 12:01 - 13:25
> 5obj 13:41 - 14:56
>=20
> Even if the overnight 1obj run was anomalous, the task would still =
take over
12
> hours to complete.=20
> Which is ok as long as you know in advance (and don't need to run it =
too many
> times).=20
> But is that a normal run time for this task? I could just have a slow
computer,
> of course.=20
>=20
> Maybe the complex region expression is a factor?
> e.g.
> expression=3D'(PATTERN<=3D12)&&(FLAG =3D=3D
0)&&(((DETX,DETY) IN
> box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
> box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
> =
circle(97,-172,17152)))&&BOX(-6435.5,13094,6633,6599,0,DETX,DETY)&&!BOX(-=
13169,-105,6599,6599,0,DETX,DETY)&&region(mos1U002-bkg_region-sky.fits)'
>=20
> I'm a little surprised with all those DETX, DETY definitions that the
> *region-sky.fits file and not the *region-det.fits file appears here.
> There are 181 regions in the region file, BTW.
>=20
> Thanks for looking into this, I look forward to your reply.
> Kind regards
> Kevin
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Gesendet: Mi 03.02.2010 14:55
> An: Briggs  Kevin Roy
> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
> =20
> Dear Kevin,
>=20
> could you please send me the Obs ID that you are using? I will try to
reproduce
> your results here. I have so far checked your problem with the SAS =
experts
and
> they tell me that it looks rather as if your computer had few CPU =
available
over
> night, especially the fact that there is so much time between the end =
of one
> task and the beginning of the next one. Could it be that a backup task =
or
> similar was running at the same time?
>=20
> Best wishes,
>=20
> Maria
>=20
> Maria Diaz Trigo
> XMM-Newton SOC
>=20
>> Full_Name: Kevin Briggs
>> Submission from: (NULL) (90.26.239.229)
>>=20
>>=20
>> Dear Helpdesk
>>=20
>> ESAS mos-spectra has been running overnight on my Mac (OSX 10.4.11 on =
a 2
GHz
>> PowerPC G5) and has only reached ccd 2. The results look ok so far =
and the
>> output log file reports no errors, but I was wondering if that is a =
normal
>> length of time required to make products for the full field. If so, =
it would
> be
>> helpful to note that in the task documentation.
>>=20
>> The command used is:
>>=20
>> mos-spectra prefix=3D"1U002" mask=3D1 elow=3D300 ehigh=3D1000 =
ccd1=3D1 ccd2=3D1 ccd3=3D1
>> ccd4=3D1 ccd5=3D1 ccd6=3D0 ccd7=3D1 =
ca

Message of length 191032 truncated


Reply 3

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Wed Feb 10 10:44:52 2010
Dear Kevin,

we have been looking at the problems that you get with ESAS here.
When running the same commands as you the make_mask runs fine. It seems that
your
machine is slow, the running time for each task is about twice as long in your
machine as in mine, but that shouldn't slow the processing so much, since in my
computer the whole task runs in minutes. It could also be that the memory is not
enough, since the process is quite memory consuming. 

However, since we do not have a definite answer, we have sent your messages to
the
ESAS experts. We will let you know as soon as we get a reply from them with a
diagnostic for your problems.

Best regards,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> This is a multi-part message in MIME format.
> 
> ------_=_NextPart_001_01CAA5E9.C6D964F0
> Content-Type: text/plain;
> 	charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
> 
> 
> Dear Maria
> 
> Please find the output log from cheese (verb=3D5) attached.
> 
> Kevin
> 
> 
> -----Urspr=FCngliche Nachricht-----
> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Gesendet: Do 04.02.2010 17:39
> An: Briggs  Kevin Roy
> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
> =20
> Dear Kevin,
> 
> I am looking in detail at your observation. The times that you report =
> are
> definitely not OK. The tasks should run within minutes. I presume that =
> the
> problems come already from the fact that 'cheese' is not running =
> properly, as
> specified in your next ticket (#43035), so let's first concentrate in =
> the other
> problem.
> 
> For this, could you please send us the full log that you get when you =
> execute
> cheese? And could you please run it with verbosity 5? I have run here =
> cheese and
> it does seem to work, though there is an issue with a big hole in the =
> output
> image which I will come back to. But first, we should understand what is
> happening when you run cheese.
> 
> Please also note that ESAS cannot be used for pn, see=20
> http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.sht=
> ml for
> more info on this issue.
> 
> Looking forward to your reply,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
> 
> 
>>=20
>> Dear Maria
>>=20
>> Many thanks for your swift reply.=20
>>=20
>> The ObsID is: 0605900101 (a proprietary data set)
>>=20
>> I will look into the possibility that a backup took up CPU overnight.
>>=20
>> The task is still running, however, now processing CCD 5.
>> Whenever I do a 'ps u', backscale seems to be running, taking 95%=20
>> of available CPU.=20
>>=20
>> I can now clarify there is no rest time between tasks. I apologise I =
> hadn't
>> looked=20
>> closely enough at the log file to spot that backscale makes several =
> calls=20
>> to other tasks, which have their start and end times before backscale
=
> ends.
>>=20
>> Instead, each backscale run on the '#obj.pi' file takes 1+ hours.=20
>>=20
>> obj 20:34 - 21:34
>> 1obj 23:36 - 06:47 (is ccd1 more complex due to the patch?)
>> 2obj 06:50 - 08:40
>> 3obj 08:57 - 11.36
>> 4obj 12:01 - 13:25
>> 5obj 13:41 - 14:56
>>=20
>> Even if the overnight 1obj run was anomalous, the task would still =
> take over
> 12
>> hours to complete.=20
>> Which is ok as long as you know in advance (and don't need to run it =
> too many
>> times).=20
>> But is that a normal run time for this task? I could just have a slow
> computer,
>> of course.=20
>>=20
>> Maybe the complex region expression is a factor?
>> e.g.
>> expression=3D'(PATTERN<=3D12)&&(FLAG =3D=3D
0)&&(((DETX,DETY) IN
>> box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
>> box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
>> =
> circle(97,-172,17152)))&&BOX(-6435.5,13094,6633,6599,0,DETX,DETY)&&!BOX(-=
> 13169,-105,6599,6599,0,DETX,DETY)&&region(mos1U002-bkg_region-sky.fits)'
>>=20
>> I'm a little surprised with all those DETX, DETY definitions that the
>> *region-sky.fits file and not the *region-det.fits file appears here.
>> There are 181 regions in the region file, BTW.
>>=20
>> Thanks for looking into this, I look forward to your reply.
>> Kind regards
>> Kevin
>>=20
>> -----Urspr=FCngliche Nachricht-----
>> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
>> Gesendet: Mi 03.02.2010 14:55
>> An: Briggs  Kevin Roy
>> Betreff: Re: ESAS mos-spectra: run 

Message of length 212100 truncated


Followup 4

Compose reply
Download message
Subject: RE: ESAS mos-spectra: run time (PR#43030)
Date: Wed, 10 Feb 2010 15:13:53 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
Dear Maria

I tried again on a Linux machine and got the same problems as with the 
Mac. That is a relatively old machine, so perhaps it has memory problems 
too. However, I got a problem with odfingest there, that I didn't get on 
the Mac and don't get if I run an older SAS version: It took much longer 
than usual and produced warnings like not finding the PN filter. These 
warnings have also been seen before and reported in PR 41432. In the 
end I ran odfingest in SASv6.1 and then progressed with the ESAS analysis
using SASv9.0, which produced the same make_mask error, and the same 
long running of mos-spectra. I didn't get as far as mos_back.

Could there be a problem with my SAS setup or CCF files, or the CIF file 
or .SAS file that leads to problems in ESAS?

Could I provide diagnostics to check if this is the case?

Many thanks
Kevin

Dear Kevin,

we have been looking at the problems that you get with ESAS here.
When running the same commands as you the make_mask runs fine. It seems that
your
machine is slow, the running time for each task is about twice as long in your
machine as in mine, but that shouldn't slow the processing so much, since in my
computer the whole task runs in minutes. It could also be that the memory is not
enough, since the process is quite memory consuming. 

However, since we do not have a definite answer, we have sent your messages to
the
ESAS experts. We will let you know as soon as we get a reply from them with a
diagnostic for your problems.

Best regards,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> This is a multi-part message in MIME format.
> 
> ------_=_NextPart_001_01CAA5E9.C6D964F0
> Content-Type: text/plain;
> 	charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
> 
> 
> Dear Maria
> 
> Please find the output log from cheese (verb=3D5) attached.
> 
> Kevin
> 
> 
> -----Urspr=FCngliche Nachricht-----
> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Gesendet: Do 04.02.2010 17:39
> An: Briggs  Kevin Roy
> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
> =20
> Dear Kevin,
> 
> I am looking in detail at your observation. The times that you report =
> are
> definitely not OK. The tasks should run within minutes. I presume that =
> the
> problems come already from the fact that 'cheese' is not running =
> properly, as
> specified in your next ticket (#43035), so let's first concentrate in =
> the other
> problem.
> 
> For this, could you please send us the full log that you get when you =
> execute
> cheese? And could you please run it with verbosity 5? I have run here =
> cheese and
> it does seem to work, though there is an issue with a big hole in the =
> output
> image which I will come back to. But first, we should understand what is
> happening when you run cheese.
> 
> Please also note that ESAS cannot be used for pn, see=20
> http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.sht=
> ml for
> more info on this issue.
> 
> Looking forward to your reply,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
> 
> 
>>=20
>> Dear Maria
>>=20
>> Many thanks for your swift reply.=20
>>=20
>> The ObsID is: 0605900101 (a proprietary data set)
>>=20
>> I will look into the possibility that a backup took up CPU overnight.
>>=20
>> The task is still running, however, now processing CCD 5.
>> Whenever I do a 'ps u', backscale seems to be running, taking 95%=20
>> of available CPU.=20
>>=20
>> I can now clarify there is no rest time between tasks. I apologise I =
> hadn't
>> looked=20
>> closely enough at the log file to spot that backscale makes several =
> calls=20
>> to other tasks, which have their start and end times before backscale =
> ends.
>>=20
>> Instead, each backscale run on the '#obj.pi' file takes 1+ hours.=20
>>=20
>> obj 20:34 - 21:34
>> 1obj 23:36 - 06:47 (is ccd1 more complex due to the patch?)
>> 2obj 06:50 - 08:40
>> 3obj 08:57 - 11.36
>> 4obj 12:01 - 13:25
>> 5obj 13:41 - 14:56
>>=20
>> Even if the overnight 1obj run was anomalous, the task would still =
> take over
> 12
>> hours to complete.=20
>> Which is ok as long as you know in advance (and don't need to run it =
> too many
>> times).=20
>> But is that a normal run time for this task? I could just have a slow
> computer,
>> of course.=20
>>=20
>> Maybe the complex region expression is a factor?
>> e.g.
>> expression=3D'(PATTERN<=3D12)&&(FLAG =3D=3D
0)&&(((DETX,DETY) IN
>> box(-2683.5,-15917,2780.5,1340,0))||((DETX,DETY) IN
>> box(2743.5,-16051,2579.5,1340,0))||((DETX,DETY) IN
>> =
> circle(97,-172,17152)))&&BO

Message of length 209862 truncated


Reply 4

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Wed Feb 10 16:13:11 2010
Dear Kevin, 

you should definitely always use the latest SAS version, i.e. 9.0 for the whole
analysis. The warning you are getting in odfingest means it is for the pn and
not for the MOSs, so definitely there should be no influence on the MOSs when
running ESAS. 

I will send you in a minute the logs that I get when running odfingest and
emchain so that you can compare them with yours. Please tell us if you find any
significant change. But please use always the latest SAS version ! We should
also sort out the reason for the failure of make_mask first with the MOSs, so I
leave pn for the moment.

Best wishes,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> 
> Dear Maria
> 
> I tried again on a Linux machine and got the same problems as with the 
> Mac. That is a relatively old machine, so perhaps it has memory problems 
> too. However, I got a problem with odfingest there, that I didn't get on 
> the Mac and don't get if I run an older SAS version: It took much longer 
> than usual and produced warnings like not finding the PN filter. These 
> warnings have also been seen before and reported in PR 41432. In the 
> end I ran odfingest in SASv6.1 and then progressed with the ESAS analysis
> using SASv9.0, which produced the same make_mask error, and the same 
> long running of mos-spectra. I didn't get as far as mos_back.
> 
> Could there be a problem with my SAS setup or CCF files, or the CIF file 
> or .SAS file that leads to problems in ESAS?
> 
> Could I provide diagnostics to check if this is the case?
> 
> Many thanks
> Kevin
> 
> Dear Kevin,
> 
> we have been looking at the problems that you get with ESAS here.
> When running the same commands as you the make_mask runs fine. It seems
that
> your
> machine is slow, the running time for each task is about twice as long in
your
> machine as in mine, but that shouldn't slow the processing so much, since
in
my
> computer the whole task runs in minutes. It could also be that the memory
is
not
> enough, since the process is quite memory consuming. 
> 
> However, since we do not have a definite answer, we have sent your messages
to
> the
> ESAS experts. We will let you know as soon as we get a reply from them with
a
> diagnostic for your problems.
> 
> Best regards,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> This is a multi-part message in MIME format.
>> 
>> ------_=_NextPart_001_01CAA5E9.C6D964F0
>> Content-Type: text/plain;
>> 	charset="iso-8859-1"
>> Content-Transfer-Encoding: quoted-printable
>> 
>> 
>> Dear Maria
>> 
>> Please find the output log from cheese (verb=3D5) attached.
>> 
>> Kevin
>> 
>> 
>> -----Urspr=FCngliche Nachricht-----
>> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
>> Gesendet: Do 04.02.2010 17:39
>> An: Briggs  Kevin Roy
>> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
>> =20
>> Dear Kevin,
>> 
>> I am looking in detail at your observation. The times that you report
=
>> are
>> definitely not OK. The tasks should run within minutes. I presume that
=
>> the
>> problems come already from the fact that 'cheese' is not running =
>> properly, as
>> specified in your next ticket (#43035), so let's first concentrate in
=
>> the other
>> problem.
>> 
>> For this, could you please send us the full log that you get when you
=
>> execute
>> cheese? And could you please run it with verbosity 5? I have run here
=
>> cheese and
>> it does seem to work, though there is an issue with a big hole in the
=
>> output
>> image which I will come back to. But first, we should understand what
is
>> happening when you run cheese.
>> 
>> Please also note that ESAS cannot be used for pn, see=20
>> http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.sht=
>> ml for
>> more info on this issue.
>> 
>> Looking forward to your reply,
>> 
>> Maria
>> 
>> Maria Diaz Trigo
>> XMM-Newton SOC
>> 
>> 
>> 
>>>=20
>>> Dear Maria
>>>=20
>>> Many thanks for your swift reply.=20
>>>=20
>>> The ObsID is: 0605900101 (a proprietary data set)
>>>=20
>>> I will look into the possibility that a backup took up CPU
overnight.
>>>=20
>>> The task is still running, however, now processing CCD 5.
>>> Whenever I do a 'ps u', backscale seems to be running, taking
95%=20
>>> of available CPU.=20
>>>=20
>>> I can now clarify there is no rest time bet

Message of length 229439 truncated


Followup 5

Compose reply
Download message
Subject: RE: ESAS mos-spectra: run time (PR#43030)
Date: Wed, 10 Feb 2010 18:20:56 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
Dear Maria

Many thanks for the logs. I'm afraid I'm away from my Mac for the next 
two days or so, so I will get back to you maybe on Friday, maybe next 
Monday.

Yes, resorting to the old SAS was a desperation measure, just to see 
if it was a dataset problem or something had changed with odfingest 
between the versions.

Thanks 
Kevin


-----Original Message-----
From: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
Sent: Wed 10.02.2010 17:13
To: Briggs  Kevin Roy
Subject: Re: ESAS mos-spectra: run time (PR#43030)
 
Dear Kevin, 

you should definitely always use the latest SAS version, i.e. 9.0 for the whole
analysis. The warning you are getting in odfingest means it is for the pn and
not for the MOSs, so definitely there should be no influence on the MOSs when
running ESAS. 

I will send you in a minute the logs that I get when running odfingest and
emchain so that you can compare them with yours. Please tell us if you find any
significant change. But please use always the latest SAS version ! We should
also sort out the reason for the failure of make_mask first with the MOSs, so I
leave pn for the moment.

Best wishes,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> 
> Dear Maria
> 
> I tried again on a Linux machine and got the same problems as with the 
> Mac. That is a relatively old machine, so perhaps it has memory problems 
> too. However, I got a problem with odfingest there, that I didn't get on 
> the Mac and don't get if I run an older SAS version: It took much longer 
> than usual and produced warnings like not finding the PN filter. These 
> warnings have also been seen before and reported in PR 41432. In the 
> end I ran odfingest in SASv6.1 and then progressed with the ESAS analysis
> using SASv9.0, which produced the same make_mask error, and the same 
> long running of mos-spectra. I didn't get as far as mos_back.
> 
> Could there be a problem with my SAS setup or CCF files, or the CIF file 
> or .SAS file that leads to problems in ESAS?
> 
> Could I provide diagnostics to check if this is the case?
> 
> Many thanks
> Kevin
> 
> Dear Kevin,
> 
> we have been looking at the problems that you get with ESAS here.
> When running the same commands as you the make_mask runs fine. It seems
that
> your
> machine is slow, the running time for each task is about twice as long in
your
> machine as in mine, but that shouldn't slow the processing so much, since
in
my
> computer the whole task runs in minutes. It could also be that the memory
is
not
> enough, since the process is quite memory consuming. 
> 
> However, since we do not have a definite answer, we have sent your messages
to
> the
> ESAS experts. We will let you know as soon as we get a reply from them with
a
> diagnostic for your problems.
> 
> Best regards,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> This is a multi-part message in MIME format.
>> 
>> ------_=_NextPart_001_01CAA5E9.C6D964F0
>> Content-Type: text/plain;
>> 	charset="iso-8859-1"
>> Content-Transfer-Encoding: quoted-printable
>> 
>> 
>> Dear Maria
>> 
>> Please find the output log from cheese (verb=3D5) attached.
>> 
>> Kevin
>> 
>> 
>> -----Urspr=FCngliche Nachricht-----
>> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
>> Gesendet: Do 04.02.2010 17:39
>> An: Briggs  Kevin Roy
>> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
>> =20
>> Dear Kevin,
>> 
>> I am looking in detail at your observation. The times that you report =
>> are
>> definitely not OK. The tasks should run within minutes. I presume that
=
>> the
>> problems come already from the fact that 'cheese' is not running =
>> properly, as
>> specified in your next ticket (#43035), so let's first concentrate in =
>> the other
>> problem.
>> 
>> For this, could you please send us the full log that you get when you =
>> execute
>> cheese? And could you please run it with verbosity 5? I have run here =
>> cheese and
>> it does seem to work, though there is an issue with a big hole in the =
>> output
>> image which I will come back to. But first, we should understand what
is
>> happening when you run cheese.
>> 
>> Please also note that ESAS cannot be used for pn, see=20
>> http://xmm.esac.esa.int/sas/current/watchout/9.0.0/pn_back_under_esas.sht=
>> ml for
>> more info on this issue.
>> 
>> Looking forward to your reply,
>> 
>> Maria
>> 
>> Maria Diaz Trigo
>> XMM-Newton SOC
>> 
>> 
>> 
>>>=20
>>> Dear Maria
>>>=20
>>> Many thanks for your swift reply.=20
>>>=20
>>> The ObsID is: 06059001

Message of length 226815 truncated


Reply 5

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Wed Mar  3 13:41:22 2010
Dear Kevin,

the SAS developers have been looking into your problem in detail and they have 
done a patch to solve the errors that were shown into your log. The software is

currently under validation and will be released with the next SAS version in
about
4 weeks time. 

If it is very urgent for you to get the results, I can ask them to run the
validation with your observation and send you the output files so that you can
continue your analysis.


Best regards,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> 
> Dear Maria
> 
> Many thanks for the logs. I'm afraid I'm away from my Mac for the next 
> two days or so, so I will get back to you maybe on Friday, maybe next 
> Monday.
> 
> Yes, resorting to the old SAS was a desperation measure, just to see 
> if it was a dataset problem or something had changed with odfingest 
> between the versions.
> 
> Thanks 
> Kevin
> 
> 
> -----Original Message-----
> From: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Sent: Wed 10.02.2010 17:13
> To: Briggs  Kevin Roy
> Subject: Re: ESAS mos-spectra: run time (PR#43030)
>  
> Dear Kevin, 
> 
> you should definitely always use the latest SAS version, i.e. 9.0 for the
whole
> analysis. The warning you are getting in odfingest means it is for the pn
and
> not for the MOSs, so definitely there should be no influence on the MOSs
when
> running ESAS. 
> 
> I will send you in a minute the logs that I get when running odfingest and
> emchain so that you can compare them with yours. Please tell us if you find
any
> significant change. But please use always the latest SAS version ! We
should
> also sort out the reason for the failure of make_mask first with the MOSs,
so
I
> leave pn for the moment.
> 
> Best wishes,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> 
>> Dear Maria
>> 
>> I tried again on a Linux machine and got the same problems as with the

>> Mac. That is a relatively old machine, so perhaps it has memory
problems 
>> too. However, I got a problem with odfingest there, that I didn't get
on 
>> the Mac and don't get if I run an older SAS version: It took much
longer 
>> than usual and produced warnings like not finding the PN filter. These

>> warnings have also been seen before and reported in PR 41432. In the 
>> end I ran odfingest in SASv6.1 and then progressed with the ESAS
analysis
>> using SASv9.0, which produced the same make_mask error, and the same 
>> long running of mos-spectra. I didn't get as far as mos_back.
>> 
>> Could there be a problem with my SAS setup or CCF files, or the CIF
file 
>> or .SAS file that leads to problems in ESAS?
>> 
>> Could I provide diagnostics to check if this is the case?
>> 
>> Many thanks
>> Kevin
>> 
>> Dear Kevin,
>> 
>> we have been looking at the problems that you get with ESAS here.
>> When running the same commands as you the make_mask runs fine. It seems
that
>> your
>> machine is slow, the running time for each task is about twice as long
in
> your
>> machine as in mine, but that shouldn't slow the processing so much,
since in
> my
>> computer the whole task runs in minutes. It could also be that the
memory is
> not
>> enough, since the process is quite memory consuming. 
>> 
>> However, since we do not have a definite answer, we have sent your
messages
> to
>> the
>> ESAS experts. We will let you know as soon as we get a reply from them
with
a
>> diagnostic for your problems.
>> 
>> Best regards,
>> 
>> Maria
>> 
>> Maria Diaz Trigo
>> XMM-Newton SOC
>> 
>>> This is a multi-part message in MIME format.
>>> 
>>> ------_=_NextPart_001_01CAA5E9.C6D964F0
>>> Content-Type: text/plain;
>>> 	charset="iso-8859-1"
>>> Content-Transfer-Encoding: quoted-printable
>>> 
>>> 
>>> Dear Maria
>>> 
>>> Please find the output log from cheese (verb=3D5) attached.
>>> 
>>> Kevin
>>> 
>>> 
>>> -----Urspr=FCngliche Nachricht-----
>>> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
>>> Gesendet: Do 04.02.2010 17:39
>>> An: Briggs  Kevin Roy
>>> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
>>> =20
>>> Dear Kevin,
>>> 
>>> I am looking in detail at your observation. The times that you
report =
>>> are
>>> definitely not OK. The tasks should run within minutes. I presume
that =
>>> the
>>> problems come alre

Message of length 246574 truncated


Followup 6

Compose reply
Download message
Subject: AW: ESAS mos-spectra: run time (PR#43030)
Date: Wed, 3 Mar 2010 16:57:28 +0100
From: "Briggs  Kevin Roy" <briggs@astro.phys.ethz.ch>
To: "Maria Diaz Trigo" <xmmhelp@sciops.esa.int>
Dear Maria

Many thanks for the update. That's great that the developers were able to 
solve the issues. Does this refer to the ESAS/mos-spectra or the odfingest 
problems?

I would like to do the analysis quite urgently if possible.
What would be the resulting files of running the validation on my observation
(a rough description, not a full list will be fine), and would I be able to
continue 
(e.g. ESAS) analysis in SASv9.0.0? Then I can decide whether it would be
worthwhile 
asking the developers to spend time on that.

With regards to the upcoming SAS version, will the ESAS software in that
version allow analysis of PN data, too?

Kind regards
Kevin


-----Urspr.ngliche Nachricht-----
Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
Gesendet: Mi 03.03.2010 14:41
An: Briggs  Kevin Roy
Betreff: Re: ESAS mos-spectra: run time (PR#43030)
 
Dear Kevin,

the SAS developers have been looking into your problem in detail and they have 
done a patch to solve the errors that were shown into your log. The software is

currently under validation and will be released with the next SAS version in
about
4 weeks time. 

If it is very urgent for you to get the results, I can ask them to run the
validation with your observation and send you the output files so that you can
continue your analysis.


Best regards,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> 
> Dear Maria
> 
> Many thanks for the logs. I'm afraid I'm away from my Mac for the next 
> two days or so, so I will get back to you maybe on Friday, maybe next 
> Monday.
> 
> Yes, resorting to the old SAS was a desperation measure, just to see 
> if it was a dataset problem or something had changed with odfingest 
> between the versions.
> 
> Thanks 
> Kevin
> 
> 
> -----Original Message-----
> From: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Sent: Wed 10.02.2010 17:13
> To: Briggs  Kevin Roy
> Subject: Re: ESAS mos-spectra: run time (PR#43030)
>  
> Dear Kevin, 
> 
> you should definitely always use the latest SAS version, i.e. 9.0 for the
whole
> analysis. The warning you are getting in odfingest means it is for the pn
and
> not for the MOSs, so definitely there should be no influence on the MOSs
when
> running ESAS. 
> 
> I will send you in a minute the logs that I get when running odfingest and
> emchain so that you can compare them with yours. Please tell us if you find
any
> significant change. But please use always the latest SAS version ! We
should
> also sort out the reason for the failure of make_mask first with the MOSs,
so
I
> leave pn for the moment.
> 
> Best wishes,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> 
>> Dear Maria
>> 
>> I tried again on a Linux machine and got the same problems as with the 
>> Mac. That is a relatively old machine, so perhaps it has memory
problems 
>> too. However, I got a problem with odfingest there, that I didn't get
on 
>> the Mac and don't get if I run an older SAS version: It took much
longer 
>> than usual and produced warnings like not finding the PN filter. These 
>> warnings have also been seen before and reported in PR 41432. In the 
>> end I ran odfingest in SASv6.1 and then progressed with the ESAS
analysis
>> using SASv9.0, which produced the same make_mask error, and the same 
>> long running of mos-spectra. I didn't get as far as mos_back.
>> 
>> Could there be a problem with my SAS setup or CCF files, or the CIF
file 
>> or .SAS file that leads to problems in ESAS?
>> 
>> Could I provide diagnostics to check if this is the case?
>> 
>> Many thanks
>> Kevin
>> 
>> Dear Kevin,
>> 
>> we have been looking at the problems that you get with ESAS here.
>> When running the same commands as you the make_mask runs fine. It seems
that
>> your
>> machine is slow, the running time for each task is about twice as long
in
> your
>> machine as in mine, but that shouldn't slow the processing so much,
since in
> my
>> computer the whole task runs in minutes. It could also be that the
memory is
> not
>> enough, since the process is quite memory consuming. 
>> 
>> However, since we do not have a definite answer, we have sent your
messages
> to
>> the
>> ESAS experts. We will let you know as soon as we get a reply from them
with
a
>> diagnostic for your problems.
>> 
>> Best regards,
>> 
>> Maria
>> 
>> Maria Diaz Trigo
>> XMM-Newton SOC
>> 
>>> This is a multi-part message in MIME format.
>>> 
>>> ------_=_NextPart_001_01CAA5E9.C6D964F0
>>> Content-Type: text/plain;
>>> 	charset="iso-8859-1"
>>> Content-Transfer-Encoding: quoted-printable
>>> 
>>&g

Message of length 244212 truncated


Reply 6

Resend
From: Maria Diaz Trigo <xmmhelp@sciops.esa.int>
To: briggs@astro.phys.ethz.ch
Subject: Re: ESAS mos-spectra: run time (PR#43030)
Date: Fri Mar 12 16:16:05 2010
Dear Kevin,

I will send you in a minute an e-mail from another account with a tar file
including all the products that I got running the following:

emchain
mos-filter 
cheese prefixm="1U002 2U002" scale=0.9 rate=1.0 dist=20.0 clobber=1 elow=300
ehigh=8000

Please check if the products are what you expect. If you agree, I will give the
number of your observation then for the SAS validation.

In any case, in two weeks the new SAS should be available. If you see any
problems in the products, please let me know as soon as possible so that they
can be corrected before SAS is released.

Many thanks in advance and again apologies for the delay.

I have asked the SAS developers about the pn as well, but they tell me that ESAS
software in version 10 will still not allow analysis of PN data.

Best wishes,

Maria

Maria Diaz Trigo
XMM-Newton SOC

> 
> Dear Maria
> 
> Many thanks for the update. That's great that the developers were able to 
> solve the issues. Does this refer to the ESAS/mos-spectra or the odfingest

> problems?
> 
> I would like to do the analysis quite urgently if possible.
> What would be the resulting files of running the validation on my
observation
> (a rough description, not a full list will be fine), and would I be able
to
> continue 
> (e.g. ESAS) analysis in SASv9.0.0? Then I can decide whether it would be
> worthwhile 
> asking the developers to spend time on that.
> 
> With regards to the upcoming SAS version, will the ESAS software in that
> version allow analysis of PN data, too?
> 
> Kind regards
> Kevin
> 
> 
> -----Urspr.ngliche Nachricht-----
> Von: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
> Gesendet: Mi 03.03.2010 14:41
> An: Briggs  Kevin Roy
> Betreff: Re: ESAS mos-spectra: run time (PR#43030)
>  
> Dear Kevin,
> 
> the SAS developers have been looking into your problem in detail and they
have

> done a patch to solve the errors that were shown into your log. The
software
is
> 
> currently under validation and will be released with the next SAS version
in
> about
> 4 weeks time. 
> 
> If it is very urgent for you to get the results, I can ask them to run the
> validation with your observation and send you the output files so that you
can
> continue your analysis.
> 
> 
> Best regards,
> 
> Maria
> 
> Maria Diaz Trigo
> XMM-Newton SOC
> 
>> 
>> Dear Maria
>> 
>> Many thanks for the logs. I'm afraid I'm away from my Mac for the next

>> two days or so, so I will get back to you maybe on Friday, maybe next 
>> Monday.
>> 
>> Yes, resorting to the old SAS was a desperation measure, just to see 
>> if it was a dataset problem or something had changed with odfingest 
>> between the versions.
>> 
>> Thanks 
>> Kevin
>> 
>> 
>> -----Original Message-----
>> From: Maria Diaz Trigo [mailto:xmmhelp@sciops.esa.int]
>> Sent: Wed 10.02.2010 17:13
>> To: Briggs  Kevin Roy
>> Subject: Re: ESAS mos-spectra: run time (PR#43030)
>>  
>> Dear Kevin, 
>> 
>> you should definitely always use the latest SAS version, i.e. 9.0 for
the
> whole
>> analysis. The warning you are getting in odfingest means it is for the
pn
and
>> not for the MOSs, so definitely there should be no influence on the
MOSs
when
>> running ESAS. 
>> 
>> I will send you in a minute the logs that I get when running odfingest
and
>> emchain so that you can compare them with yours. Please tell us if you
find
> any
>> significant change. But please use always the latest SAS version ! We
should
>> also sort out the reason for the failure of make_mask first with the
MOSs,
so
> I
>> leave pn for the moment.
>> 
>> Best wishes,
>> 
>> Maria
>> 
>> Maria Diaz Trigo
>> XMM-Newton SOC
>> 
>>> 
>>> Dear Maria
>>> 
>>> I tried again on a Linux machine and got the same problems as with
the 
>>> Mac. That is a relatively old machine, so perhaps it has memory
problems 
>>> too. However, I got a problem with odfingest there, that I didn't
get on 
>>> the Mac and don't get if I run an older SAS version: It took much
longer 
>>> than usual and produced warnings like not finding the PN filter.
These 
>>> warnings have also been seen before and reported in PR 41432. In
the 
>>> end I ran odfingest in SASv6.1 and then progressed with the ESAS
analysis
>>> using SASv9.0, which produced the same make_mask error, and the
same 
>>> long running of mos-spectra. I didn't get as far as mos_back.
>>> 
>>> Cou

Message of length 264698 truncated

Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page