Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing Data/34135
Full headers

From: "Antonino D'Ai'" <dai@fisica.unipa.it>
Subject: Information request on OBS.ID. 0505950101
Compose reply
Download message
Move To:
4 replies: 1 2 3 4
2 followups: 1 2

Private message: yes  no

Notes:

Notification:


Date: Fri, 16 Nov 2007 12:57:48 +0100
From: "Antonino D'Ai'" <dai@fisica.unipa.it>
To: xmmhelp@sciops.esa.int
Subject: Information request on OBS.ID. 0505950101
Dear XMM-SOC,

I have enountered some problems regarding the download of some PPS datasets.
On Wednesday, a notification e-mail stated the availability  of PPS products for

XMM-Newton Observation Id : 0505950101

XMM-Newton pipeline processing report:

Target name          : GX 340+0
Observer             : Dr Antonino D'Ai
Observation date     : Sep 2 2007 12:58PM
Observation duration : 48912
Processing date      : Nov 6 2007 3:07PM

but among the datasets, I do not find any eventlist file, and any other related
file, associated with the Epic-PN instrument (which was set in Timing Mode).

Could you, please, report me about the cause of this ? Are PN data still under
process ?

I thank you in advace.

My best regards

Antonino D'A.. 


Reply 1

Resend
From: Matteo Guainazzi <xmmhelp@sciops.esa.int>
To: dai@fisica.unipa.it
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Date: Fri Nov 16 16:57:54 2007
Dear Antonino,

 there is nothing wrong with the pn data. However, the event files are simply
too large for the current hardware configuration at SSC to eb able to process
them. Hence, the decision was taken to skip processing of the pn exposures, not
to unduly delay the data delivery.

 There are two possible ways to address this issue:

1) you may try to reduce the data yourself, if you have access good hardware
facilities. Be advised in this case to try SAS in "low memory mode" if the first
attempt with the standard "high memory mode" fails.

2) the SOC may ask for specific processing of the pn exposure to the pn
Instrument Team. In this case, pn event list would be made available to you via
ftp account.

 Please, let me know which of the above ways above you would like to follow.

 I apologize for any inconvenience.

 Regards, Matteo


Followup 1

Compose reply
Download message
Date: Tue, 27 Nov 2007 12:38:44 +0100
From: "Antonino D'Ai'" <dai@fisica.unipa.it>
To: Matteo Guainazzi <xmmhelp@sciops.esa.int>
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Dear Matteo, 

We tried to reduce the pn data without success on some well equipped
machines (a biproc AMD64 opteron 254 machine with 4 GB of RAM and 4 GB of
swap). Even using the low or high/low memory modes the extraction fails
during the epevents process depending from run to run. 

First of all, we are using the latest software version:
sasversion sasversion:- XMM-Newton SAS release and build information: 
SAS release: xmmsas_20070708_1801-7.1.0 Compiled on: Sun Jul 8 18:07:52 GMT
2007 Compiled by: sasbuild@xmmls11 
Platform : Red Hat Linux release 9 (Shrike) SAS-related environment
variables that are set: SAS_DIR = /opt/xmmsas_20070708_1801 SAS_PATH =
/opt/xmmsas_20070708_1801

Here is the output of one epchain run:

epchain datamode=TIMING ccds=4 memorymodel=low EXPOSURE = 1
/usr/users/lavaget/0505950101/odf//1416_0505950101_PNS00304TIE.FIT 
** epframes: warning (BPTcode), BadPixelTable code differs in PAH.FITS
[2_000_BAD] and ADUCONV.CCF [0_002_BAD] 
** epframes: warning (wrongReadoutOrder), Events in wrong order: Event
21680663 : IX,IY = 30 2 ; Previous event: IX,IY = 8 118 
** epframes: warning (wrongReadoutOrder), Events in wrong order: Event
44930208 : IX,IY = 3 2 ; Previous event: IX,IY = 46 150 
** epframes: warning (wrongReadoutOrder), Events in wrong order: Event
44930244 : IX,IY = 41 62 ; Previous event: IX,IY = 36 168 badpixfind:-
Executing (routine): badpixfind eventset=rawevents04.dat lothresh=0
hithresh=0.00025 searchbadpix=yes columnsearchlabel=median locolthresh=0
hicolthresh=7e-05 withsearchbadcolumn=yes rawxlist='' rawylist=''
typelist='' yextentlist='' userflagbadpix=no thresholdlabel=rate
flickertimesteps=1 flickerksthresh=0.55 flickerchisqthresh=15 cellsize=2
backgroundrate=-1 narrowerthanpsf=0 badpixset=badpixfind.fits
badpixmap=P0505950101PNS003BPXHMK0004.FIT withbadpixmap=yes withfovmask=yes
threshabovebackground=yes mappixcellsize=1 loenergythresh=7
hienergythresh=15 ccd=-1 -w 10 -V 3 
** badpixfind: warning (findRealSources), chosen narrowerthanpsf parameter
very low - real sources may be found badpixfind:- Executing (routine):
badpixfind eventset=rawevents04.dat lothresh=0 hithresh=0.003
searchbadpix=yes columnsearchlabel=median locolthresh=0 hicolthresh=0.0012
withsearchbadcolumn=yes rawxlist='' rawylist='' typelist='' yextentlist=''
userflagbadpix=no thresholdlabel=rate flickertimesteps=1
flickerksthresh=0.55 flickerchisqthresh=15 cellsize=2 backgroundrate=0.0001
narrowerthanpsf=0 badpixset=badpixfind.fits
badpixmap=P0505950101PNS003BPXMSK0004.FIT withbadpixmap=yes withfovmask=no
threshabovebackground=no mappixcellsize=1 loenergythresh=0.16
hienergythresh=10 ccd=-1 -w 10 -V 3 
** badpixfind: warning (findRealSources), chosen narrowerthanpsf parameter
very low - real sources may be found 
** badpixfind: warning (noGoodPixsFound), No good pixels found - thresholds
OK? Cannot allocate array temporary - out of memory Program terminated by
fatal error Then the rest of the chain fails... 
** attcalc: error (badInfile), Input file incorrect - no DETX column 
** evselect: error (UnknownColumn), No column named `PI' in table
`cleanevents04.dat:EVENTS' Column `PI' does not exist in table
`cleanevents04.dat:EVENTS' ** evlistcomb: warning (sortFiles14),
merged.tim:OFFSETS is empty ** -: error (constituent), One (or more) of the
constituent tasks ended in error ! 

The problem is - when I monitor the memory usage - that no process exceeds
2.4 GB of used RAM, much less than the RAM alone, no matter the RAM+swap
grand total! I checked for the existence of limits in memory usage: 

$ ulimit -a core file size (blocks, -c) 0 
data seg size (kbytes, -d) unlimited 
max nice (-e) 20 file size (blocks, -f) unlimited 
pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited 
max memory size (kbytes, -m) unlimited 
open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues
(bytes, -q) unlimited 
max rt priority (-r) unlimited 
stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited 
max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited 
file locks (-x) unlimited


It seems to me, therefore, a software issue rather than an hardware issue.
I have noticed that in the past other people dealing with large datasets
had similar problems (see request SASv6.0/10847 by Dr Jon M. Miller and
SASv7.0/26219 by Dr Sundip Bhattacharyya), but in both cases no solution is
reported. Is there any way to deal with this issue correctly? Or can the PN
team help us in correctly extracting the data ?

Please let us know as soon as possible. 

Best Regards 

Antonino D'Ai' & Giuseppe Lavagetto

DSFA, Palermo

On Fri, 16 Nov 2007 16:57:53 GMT, Matteo Guainazzi
<xmmhelp@sciops.esa.int>
wrote:
> Dear Antonino,
> 
>  there is nothing wrong with the pn data. However, the event files are
> simply
> too large for the current hardware configuration at SSC to eb able to
> process
> them. Hence, the decision was taken to skip processing of the pn
> 

Message of length 6407 truncated


Reply 2

Resend
From: Matteo Guainazzi <xmmhelp@sciops.esa.int>
To: dai@fisica.unipa.it
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Date: Tue Nov 27 16:22:35 2007
Dear Antonino, dear Giuseppe,

> Please let us know as soon as possible. 

 First of all I would like to warmly thank you for your investigations and your
report. I have transmitted it to the SAS developers, who are investigating this
issue.

 We can try and reprocess the pn event lists for you. In previous cases this has
worked.

 I will let you know where you can download the event lists from, once they are
ready.

 Let me again apologize for any inconvenience.

 Regards, Matteo


Reply 3

Resend
From: Matteo Guainazzi <xmmhelp@sciops.esa.int>
To: dai@fisica.unipa.it
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Date: Tue Nov 27 16:50:04 2007
Dear Antonino,

 may I dare asking you another favor? Could you please send me a SAS_VERBOSITY=5
log of a failing "epchain" or "epproc" ("epframes" may be sufficient, assuming
that this task is the failing one as it eats/needs the memory). This would help
us.

 Thanks, Matteo


Reply 4

Resend
From: Matteo Guainazzi <xmmhelp@sciops.esa.int>
To: dai@fisica.unipa.it
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Date: Wed Nov 28 11:02:24 2007
Dear Antonino,

 the pn event list of the subject observation is available at:

ftp://ftp.xray.mpe.mpg.de/people/mjf/.rvEtvwer9n8/.vtr1QQ34./P0505950101PNS003TIEVLI0000.FIT.gz

 Please, let me know when you have downloaded it, and made some basic checks.

 Thanks, Matteo


Followup 2

Compose reply
Download message
Date: Wed, 28 Nov 2007 17:48:41 +0100
From: "Antonino D'Ai'" <dai@fisica.unipa.it>
To: Matteo Guainazzi <xmmhelp@sciops.esa.int>
Subject: Re: Information request on OBS.ID. 0505950101 (PR#34135)
Dear Matteo, 

I've downloaded the data and extracted the spectrum (!!!), 
well, it seems, as first look, that everything is ok. 

Thank You very very much for your kind help and quick assistance :) 

My Best Regards 

Antonino 



On Wed, 28 Nov 2007 11:02:24 GMT, Matteo Guainazzi
<xmmhelp@sciops.esa.int>
wrote:
> Dear Antonino,
> 
>  the pn event list of the subject observation is available at:
> 
>
ftp://ftp.xray.mpe.mpg.de/people/mjf/.rvEtvwer9n8/.vtr1QQ34./P0505950101PNS003TIEVLI0000.FIT.gz
> 
>  Please, let me know when you have downloaded it, and made some basic
> checks.
> 
>  Thanks, Matteo
> 
> 
>
================================================================================================
> This message and any attachments are intended for the use of the
addressee
> or addressees only. The
> unauthorised disclosure, use, dissemination or copying (either in whole
or
> in part) of its content
> is prohibited. If you received this message in error, please delete it
> from your system and notify
> the sender. E-mails can be altered and their integrity cannot be
> guaranteed. ESA shall not be liable
> for any e-mail if modified.
>
=================================================================================================


Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page