Up to top level
AO15   AO16   AO17   AO18   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing esas/84804
Full headers

From: idroberts080@gmail.com
Subject: Problem extracting spectrum from region using mos-spectra
Compose reply
Download message
Move To:
3 replies: 1 2 3
2 followups: 1 2

Private message: yes  no

Notes:

Notification:


Date: Mon, 25 Mar 2019 19:47:50 +0100
From: idroberts080@gmail.com
To: xmmhelp@sciops.esa.int
Subject: Problem extracting spectrum from region using mos-spectra
Full_Name: Ian Roberts
Submission from: (NULL) (193.147.152.102)


I am having trouble extracting a spectrum from a region (defined in detector
coordinates) using mos-spectra.  mos-spectra runs to completion however when I
look at 'mos2S002-obj-im-500-7000.fits' I find that it includes the entire mos2
detector and not just the region that I have specified.  I am using SAS release
xmmsas_20170112_1337-16.0.0 and am analysing obsID 0153750101.  The command
chain I am using is the following:

- Download ODF data, run cifbuild and odfingest, set $SAS_ODF and $SAS_CCF to
point to the *.SAS and *.ccf files respecively

- Run emchain

- Run mos-filter

- Run cheese prefixm="1S001 2S002" scale=0.25 rate=1.0 dist=40.0 clobber=1
elow=500 ehigh=7000

- Run mos-spectra prefix=2S002 caldb=$SAS_CCFPATH region=poly0.txt mask=1
elow=500 ehigh=7000 ccd1=1 ccd2=1 ccd3=1 ccd4=1 ccd5=1 ccd6=1 ccd7=1

I am trying to extract a spectra from a polygon region which is defined in
poly0.txt.  The file contains the following line:

&&((DETX,DETY) IN
polygon(4270.5,2752.4,4687.1,936.0,5429.1,3366.4,5997.6,888.4))

I've also tried extracting a spectrum from a circular region as a sanity check
and run into the exact same issue.  I defined the circular regions as:

&&((DETX,DETY) IN circle(4270.5,2752.4,1800))

I'd appreciate any help that you can offer.

Thanks!
Ian


Reply 1

Resend
From: Jacobo Ebrero <xmmhelp@sciops.esa.int>
To: idroberts080@gmail.com
Subject: Re: Problem extracting spectrum from region using mos-spectra (PR#84804)
Date: Tue Mar 26 17:00:20 2019
Dear Ian,

Could you please send me the full list of commands you have used, including the
SAS start-up and also those for the analysis of MOS1?

I understand that this issue only shows up during the MOS2 analysis, but not for
MOS1. Also please make sure that, in the call to mos-spectra, the parameter
caldb points to where the CalDB files required to run ESAS are located.

On a side note, it is always advisable to run the latest version of SAS
(currently v17) but this is possibly unrelated to your problem.

Best regards,
Jacobo
----------------------
Dr. Jacobo Ebrero
XMM-Newton SOC
Community Support and Scientific Planning Team


Followup 1

Compose reply
Download message
From: Ian Roberts <idroberts080@gmail.com>
Date: Tue, 26 Mar 2019 12:19:16 -0400
Subject: Re: Problem extracting spectrum from region using mos-spectra (PR#84804)
To: Jacobo Ebrero <xmmhelp@sciops.esa.int>
--0000000000005b80cd058501af16
Content-Type: text/plain; charset="UTF-8"

Hi Jacobo:

Sorry I should have made this clear, but this seems to be an issue for me
for both MOS1 and MOS2, I was just using MOS2  as an example to demonstrate
the problem.

Here is my full chain of commands.

0. The ODF files for the observation are downloaded from here and then
unzipped:  https://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/w3browse.pl

1. sasinit

2.  export SAS_ODF=/2/home/idroberts/coma/xmm/0153750101/ODF

3.  mkdir analysis

4.  cd analysis

5.  cifbuild

6.  export SAS_CCF=/2/home/idroberts/coma/xmm/0153750101/analysis/ccf.cif

7.  odfingest odfdir=$SAS_ODF outdir=$SAS_ODF

8.  export
SAS_ODF=/2/home/idroberts/coma/xmm/0153750101/ODF/0364_0153750101_SCX00000SUM.SAS

9.  emchain

10.  mos-filter

11.  cheese prefixm="1S001 2S002" scale=0.25 rate=1.0 dist=40.0 clobber=1
elow=500 ehigh=7000

12a.  mos-spectra prefix=2S002 caldb=$SAS_CCFPATH region=poly0.txt mask=1
elow=500 ehigh=7000 ccd1=1 ccd2=1 ccd3=1 ccd4=1 ccd5=1 ccd6=1 ccd7=1

or for MOS1

12b.  mos-spectra prefix=1S001 caldb=$SAS_CCFPATH region=poly0.txt mask=1
elow=500 ehigh=7000 ccd1=1 ccd2=1 ccd3=1 ccd4=1 ccd5=0 ccd6=1 ccd7=1

where poly0.reg contains the polygon region in detector coordinates that I
specified in the last message.

Let me know if you require any further information from me.

Thanks,
Ian

On Tue, Mar 26, 2019 at 12:00 PM Jacobo Ebrero <xmmhelp@sciops.esa.int>
wrote:

> Dear Ian,
>
> Could you please send me the full list of commands you have used,
> including the
> SAS start-up and also those for the analysis of MOS1?
>
> I understand that this issue only shows up during the MOS2 analysis, but
> not for
> MOS1. Also please make sure that, in the call to mos-spectra, the parameter
> caldb points to where the CalDB files required to run ESAS are located.
>
> On a side note, it is always advisable to run the latest version of SAS
> (currently v17) but this is possibly unrelated to your problem.
>
> Best regards,
> Jacobo
> ----------------------
> Dr. Jacobo Ebrero
> XMM-Newton SOC
> Community Support and Scientific Planning Team
>
> This message is intended only for the recipient(s) named above. It may
> contain proprietary information and/or
> protected content. Any unauthorised disclosure, use, retention or
> dissemination is prohibited. If you have received
> this e-mail in error, please notify the sender immediately. ESA applies
> appropriate organisational measures to protect
> personal data, in case of data privacy queries, please contact the ESA
> Data Protection Officer (dpo@esa.int).
>
>
>

--0000000000005b80cd058501af16
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div
dir=3D"ltr"><div>Hi=
 Jacobo:</div><div><br></div><div>Sorry I should
have made this clear, but =
this seems to be an issue for me for both MOS1 and MOS2, I was just using M=
OS2=C2=A0 as an example to demonstrate the
problem.</div><div><br></div><di=
v>Here is my full chain of
commands.</div><div><br></div><div>0. The ODF fi=
les for the observation are downloaded from here and then unzipped:=C2=A0 <=
a href=3D"https://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/w3browse.pl">https=
://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/w3browse.pl</a></div><div><br></d=
iv><div>1. sasinit<br></div><div><br></div><div>2.=C2=A0
export SAS_ODF=3D/=
2/home/idroberts/coma/xmm/0153750101/ODF</div><div><br></div><div>3.=C2=A0
=
mkdir analysis</div><div><br></div><div>4.=C2=A0
cd analysis</div><div><br>=
</div><div>5.=C2=A0
cifbuild</div><div><br></div><div>6.=C2=A0 export
SAS_C=
CF=3D/2/home/idroberts/coma/xmm/0153750101/analysis/ccf.cif</div><div><br><=
/div><div>7.<tt>=C2=A0 </tt>odfingest odfdir=3D$SAS_ODF
outdir=3D$SAS_ODF</=
div><div><br></div><div>8.=C2=A0 export
SAS_ODF=3D/2/home/idroberts/coma/xm=
m/0153750101/ODF/0364_0153750101_SCX00000SUM.SAS</div><div><br></div><div>9=
.=C2=A0 emchain</div><div><br></div><div>10.=C2=A0
mos-filter</div><div><br=
></div><div>11.=C2=A0 cheese prefixm=3D&quot;1S001
2S002&quot; scale=3D0.25=
 rate=3D1.0 dist=3D40.0 clobber=3D1=20
elow=3D500 ehigh=3D7000</div><div><br></div><div>12a.=C2=A0
mos-spectra pre=
fix=3D2S002 caldb=3D$SAS_CCFPATH=20
region=3Dpoly0.txt mask=3D1 elow=3D500=20
ehigh=3D7000 ccd1=3D1 ccd2=3D1 ccd3=3D1=20
ccd4=3D1 ccd5=3D1 ccd6=3D1
ccd7

Message of length 7703 truncated


Reply 2

Resend
From: Jacobo Ebrero <xmmhelp@sciops.esa.int>
To: idroberts080@gmail.com
Subject: Re: Problem extracting spectrum from region using mos-spectra (PR#84804)
Date: Wed Mar 27 16:57:35 2019
Hi Ian,

I'm consulting with the ESAS developers. It might be possible that the region is
taken into account (for e.g. spectral extraction) but it is not reflected in the
output images. As soon as I hear back from them and can confirm this I will get
back to you.

In the meanwhile, in order to check whether the region file is correctly
accounted for, could you please send me the output log of mos-spectra?

Additionally you can also check the header of the output image file, to see if
the region information is written in there. If that is the case you can still
proceed with the spectral analysis even if the extraction region is not shown in
the image, as ESAS will take the information from the fits header.

Regards,
Jacobo

----------------------
Dr. Jacobo Ebrero
XMM-Newton SOC
Community Support and Scientific Planning Team


Followup 2

Compose reply
Download message
From: Ian Roberts <idroberts080@gmail.com>
Date: Thu, 28 Mar 2019 17:38:46 -0400
Subject: Re: Problem extracting spectrum from region using mos-spectra (PR#84804)
To: Jacobo Ebrero <xmmhelp@sciops.esa.int>
--000000000000b606fe05852e6125
Content-Type: multipart/alternative; boundary="000000000000b606fb05852e6123"

--000000000000b606fb05852e6123
Content-Type: text/plain; charset="UTF-8"

Hi Jacobo:

I have done some further testing and it seems that the issue is related to
the fact that I am calling the mos-spectra function using the 'subprocess'
module in python.  When I call mos-spectra from the terminal without using
python then everything seems to work fine.  I have attached the output from
mos-spectra both using python (_Py) and not using python (_noPy) for your
reference.  The main differences between the two that I can see is that
these two lines are present when using python:

Use of uninitialized value $FOVdef in scalar chomp at
/usr/local/SAS/xmmsas_20170112_1337/bin/mos-spectra line 386.
Use of uninitialized value $FOVdef in concatenation (.) or string at
/usr/local/SAS/xmmsas_20170112_1337/bin/mos-spectra line 394.

However they are not present when I run mos-spectra from the terminal.
Furthermore, mos-spectra recognizes the Area selection from the region file
when I run it from the terminal, but it does not when I use 'subprocess' to
call the function.

Do you know what the above errors that I am getting mean and if there is
any fix?  I would really like to call everything using python as it allows
me to automate things easily using scripts.  Interestingly, all of the
other esas functions that I use in my command chain seem to work fine when
called using 'subprocess'.

Thanks for any help.
Ian

On Wed, Mar 27, 2019 at 11:58 AM Jacobo Ebrero <xmmhelp@sciops.esa.int>
wrote:

> Hi Ian,
>
> I'm consulting with the ESAS developers. It might be possible that the
> region is
> taken into account (for e.g. spectral extraction) but it is not reflected
> in the
> output images. As soon as I hear back from them and can confirm this I
> will get
> back to you.
>
> In the meanwhile, in order to check whether the region file is correctly
> accounted for, could you please send me the output log of mos-spectra?
>
> Additionally you can also check the header of the output image file, to
> see if
> the region information is written in there. If that is the case you can
> still
> proceed with the spectral analysis even if the extraction region is not
> shown in
> the image, as ESAS will take the information from the fits header.
>
> Regards,
> Jacobo
>
> ----------------------
> Dr. Jacobo Ebrero
> XMM-Newton SOC
> Community Support and Scientific Planning Team
>
> This message is intended only for the recipient(s) named above. It may
> contain proprietary information and/or
> protected content. Any unauthorised disclosure, use, retention or
> dissemination is prohibited. If you have received
> this e-mail in error, please notify the sender immediately. ESA applies
> appropriate organisational measures to protect
> personal data, in case of data privacy queries, please contact the ESA
> Data Protection Officer (dpo@esa.int).
>
>
>

--000000000000b606fb05852e6123
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div>Hi
Jacobo:</div><div><br></div><div>=
I have done some further testing and it seems that the issue is related to =
the fact that I am calling the mos-spectra function using the &#39;subproce=
ss&#39; module in python.=C2=A0 When I call mos-spectra from the terminal w=
ithout using python then everything seems to work fine.=C2=A0 I have attach=
ed the output from mos-spectra both using python (_Py) and not using python=
 (_noPy) for your reference.=C2=A0 The main differences between the two tha=
t I can see is that these two lines are present when using
python:</div><di=
v><br></div><div>Use of uninitialized value $FOVdef in
scalar chomp at /usr=
/local/SAS/xmmsas_20170112_1337/bin/mos-spectra line 386.<br>Use of
uniniti=
alized value $FOVdef in concatenation (.) or string at /usr/local/SAS/xmmsa=
s_20170112_1337/bin/mos-spectra line
394.</div><div><br></div><div>However =
they are not present when I run mos-spectra from the terminal.=C2=A0 Furthe=
rmore, mos-spectra recognizes the Area selection from the region file when =
I run it from the terminal, but it does not when I use
&#39;subprocess&#39;=
 to call the function.</div><div><br></div><div>Do
you know what the above =
errors that I am getting mean and if there is any fix?=C2=A0 I would really=
 like to call everything using python as it allows me to automate things ea=
sily using scripts.=C2=A0 Interestingly, all of the other esas functions th=
at I use in my command chain seem to work fine when called using &#39;subpr=
ocess&#39;.</div><div><br></div><div>Thanks

Message of length 448146 truncated


Reply 3

Resend
From: Jacobo Ebrero <xmmhelp@sciops.esa.int>
To: idroberts080@gmail.com
Subject: Re: Problem extracting spectrum from region using mos-spectra (PR#84804)
Date: Fri Mar 29 12:06:13 2019
Hi Ian,

It looks like when you execute mos-spectra from python it fails to read the
contents of your region file, which is done successfully in the execution from
the command line. You can tell that by checking at the beginning of your output
logs the line starting by "Area Selection". In one the contents of the region
file are shown whereas in the other the line is blank. So it finds the file (the
error message would be different otherwise) but it seems that it is empty.

So, the question is, is the region file created from some other script that
didn't have time to finish before the call to mos-spectra?

Our recommendation, if you would like to keep executing SAS tasks via python, is
that you make sure that all the tasks are called sequentially, letting them
finish to completion before starting to execute a new one.

We hope that this helps.

Regards,
Jacobo

----------------------
Dr. Jacobo Ebrero
XMM-Newton SOC
Community Support and Scientific Planning Team

Up to top level
AO15   AO16   AO17   AO18   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page