Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing Backgrounds/85684
Full headers

From: jjung11@alaska.edu
Subject: evqpb spectrum shows more counts than science spectrum
Compose reply
Download message
Move To:
3 replies: 1 2 3
1 followups: 1

Private message: yes  no

Notes:

Notification:


Date: Wed, 25 Sep 2019 01:34:13 +0200
From: jjung11@alaska.edu
To: xmmhelp@sciops.esa.int
Subject: evqpb spectrum shows more counts than science spectrum
Full_Name: Jaewoong Jung
Submission from: (NULL) (193.147.152.102)


Dear XMM helpdesk team,

I'm doing the background study, so I removed astrophysical sources from the
observation data.
And made a spectrum from large region, circle (-50,-180,16000), with both
science exposure and FWC exposure generated by evqpb. (I also removed same
region from the FWC files)
And I loaded spectrums (science and FWC) into Xspec. 
When I loaded FWC spectrum as background, I got negative count.
I feel like evqpb does not properly account particle background, overestimating
it.
Can you tell me what may be the problem?

The event I studied was 0500070601.
The SAS commands I used were

cifbuild
odfingest
emproc

#Proton flaring filtering
evselect table=1470_0500070601_EMOS1_S001_ImagingEvts.ds withrateset=Y
rateset=rateMOS1.fits maketimecolumn=Y timebinsize=100 makeratecolumn=Y
expression='#XMMEA_EM && (PI>10000) && (PATTERN==0)'
dsplot table=rateMOS1.fits x=TIME y=RATE
tabgtigen table=rateMOS1.fits expression='RATE<=.35' gtiset=MOS1gti.fits
evselect table=1470_0500070601_EMOS1_S001_ImagingEvts.ds withfilteredset=Y
filteredset=MOS1clean.fits destruct=Y keepfilteroutput=T expression="#XMMEA_EM
&& gti(MOS1gti.fits,TIME) && (PI>150)"
evselect table=MOS1clean.fits withrateset=Y rateset=rateMOS1clean.fits
maketimecolumn=Y timebinsize=100 makeratecolumn=Y expression='#XMMEA_EM
&&
(PI>10000) && (PATTERN==0)'
dsplot table=rateMOS1clean.fits x=TIME y=RATE

#Sources finding
evselect table=MOS1clean.fits imagebinning=binSize
imageset=MOS1_image_full.fits
withimageset=y xcolumn=X ycolumn=Y ximagebinsize=40 yimagebinsize=40
expression='#XMMEA_EM && (PI in [200:12000]) && (PATTERN in
[0:12]) &&
(FLAG==0)'
edetect_chain imagesets=MOS1_image_full.fits eventsets=MOS1clean.fits
attitudeset=1470_0500070601_AttHk.ds pimin=200 pimax=12000 ecf=2.0
eboxl_list=MOS1_eboxlist_l.fits eboxm_list=MOS1_eboxlist_m.fits
esp_nsplinenodes=16 eml_list=MOS1_emllist.fits esen_mlmin=15
srcdisplay boxlistset=MOS1_emllist.fits imageset=MOS1_image_full.fits
sourceradius=.01

#Sources removal
region -w 0 -V 0 eventset=MOS1clean.fits operationstyle=global
radiusstyle=userfixed fixedradius=35 srclisttab=MOS1_emllist.fits
expression='ID_INST==0' outunit=detxy bkgregionset=MOS1reg.fits
    evselect -w 0 -V 0 table=MOS1clean.fits
expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1out.fits
updateexposure=N writedss=Y withfilteredset=Y destruct=Y keepfilteroutput=T
      ftcopy "MOS1_emllist.fits[SRCLIST][#row <= 2 ]" MOS1_emllist2.fits
clobber=yes
      region -w 0 -V 0 eventset=MOS1clean.fits operationstyle=global
radiusstyle=userfixed fixedradius=300 srclisttab=MOS1_emllist2.fits
expression='ID_INST==0' outunit=detxy bkgregionset=MOS1reg2.fits
      evselect -w 0 -V 0 table=MOS1clean.fits
expression="region(MOS1reg2.fits,DETX,DETY)" filteredset=MOS1out.fits
updateexposure=N writedss=Y withfilteredset=Y destruct=Y keepfilteroutput=T
      evselect -w 0 -V 0 table=MOS1out.fits
expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1cleanf.fits
updateexposure=N writedss=Y withfilteredset=Y destruct=Y keepfilteroutput=T
  

#spectrum generation
evselect table=MOS1cleanf.fits withspectrumset=y spectrumset=MOS1spec.fits
energycolumn=PI spectralbinsize=5 withspecranges=y specchannelmin=0
specchannelmax=11999 expression='#XMMEA_EM && (PATTERN<=12)
&& ((DETX,DETY) in
circle (-50,-180,16000))' withfilteredset=y filteredset=MOS1cleanfs.fits
backscale spectrumset=MOS1spec.fits badpixlocation=MOS1cleanfs.fits
rmfgen spectrumset=MOS1spec.fits rmfset=MOS1.rmf
arfgen spectrumset=MOS1spec.fits arfset=MOS1.arf withrmfset=y rmfset=MOS1.rmf
badpixlocation=MOS1cleanfs.fits detmaptype=flat extendedsource=y

#FWC file and spectrum generation
evqpb table=1470_0500070601_EMOS1_S001_ImagingEvts.ds exposurefactor=2.0
attfile=1470_0500070601_AttHk.ds outset=evqpb_MOS1.fits
evselect table=evqpb_MOS1.fits withfilteredset=y
filteredset=MOS1_qpb_clean.fits
destruct=y keepfilteroutput=T expression="gti(MOS1gti.fits,TIME)"
      evselect -w 0 -V 0 table=MOS1_qpb_clean.fits
expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1out_fwc.fits
updateexposure=N writedss=Y withfilteredset=Y destruct=Y keepfilteroutput=T
      evselect -w 0 -V 0 table=MOS1_qpb_clean.fits
expression="region(MOS1reg2.fits,DETX,DETY)" filteredset=MOS1_qpb_cleanf.fits
updateexposure=N writedss=Y withfilteredset=Y destruct=Y keepfilteroutput=T

evselect table=MOS1_qpb_cleanf.fits withspectrumset=y
spectrumset=MOS1fwc_spec.fits energycolumn=PI spectralbinsize=5
withspecranges=y
specchannelmin=0 specchannelmax=11999 expression='#XMMEA_EM &&
(PATTERN<=12) &&
((DETX,DETY) in circle (-50,-180,16000))' withfilteredset=y
filteredset=MOS1_qpb_cleanfs.fits zcolumn=EWEIGHT
backscale spectrumset=MOS1fwc_spec.fits badpixlocation=MOS1_qpb_cleanfs.fits
fparkey none MOS1spec.fits BACKFILE add=yes
fparkey none

Message of length 5470 truncated

Reply 1

Resend
From: Ignacio de la Calle <xmmhelp@sciops.esa.int>
To: jjung11@alaska.edu
Subject: Re: evqpb spectrum shows more counts than science spectrum (PR#85684)
Date: Thu Sep 26 11:02:19 2019
Dear Jaewoong Jung,

We are looking into your problem and will come back to you as soon as we have
an answer.

Thanks,

Ignacio de la Calle
XMM-Newton SOC

> Submission from: (NULL) (193.147.152.102)
> 
> 
> Dear XMM helpdesk team,
> 
> I'm doing the background study, so I removed astrophysical sources from
the
> observation data.
> And made a spectrum from large region, circle (-50,-180,16000), with both
> science exposure and FWC exposure generated by evqpb. (I also removed same
> region from the FWC files)
> And I loaded spectrums (science and FWC) into Xspec. 
> When I loaded FWC spectrum as background, I got negative count.
> I feel like evqpb does not properly account particle background,
overestimating
> it.
> Can you tell me what may be the problem?
> 
> The event I studied was 0500070601.
> The SAS commands I used were
> 
> cifbuild
> odfingest
> emproc
> 
> #Proton flaring filtering
> evselect table=1470_0500070601_EMOS1_S001_ImagingEvts.ds withrateset=Y
> rateset=rateMOS1.fits maketimecolumn=Y timebinsize=100 makeratecolumn=Y
> expression='#XMMEA_EM && (PI>10000) && (PATTERN==0)'
> dsplot table=rateMOS1.fits x=TIME y=RATE
> tabgtigen table=rateMOS1.fits expression='RATE<=.35'
gtiset=MOS1gti.fits
> evselect table=1470_0500070601_EMOS1_S001_ImagingEvts.ds withfilteredset=Y
> filteredset=MOS1clean.fits destruct=Y keepfilteroutput=T
expression="#XMMEA_EM
> && gti(MOS1gti.fits,TIME) && (PI>150)"
> evselect table=MOS1clean.fits withrateset=Y rateset=rateMOS1clean.fits
> maketimecolumn=Y timebinsize=100 makeratecolumn=Y expression='#XMMEA_EM
> &&
> (PI>10000) && (PATTERN==0)'
> dsplot table=rateMOS1clean.fits x=TIME y=RATE
> 
> #Sources finding
> evselect table=MOS1clean.fits imagebinning=binSize
> imageset=MOS1_image_full.fits
> withimageset=y xcolumn=X ycolumn=Y ximagebinsize=40 yimagebinsize=40
> expression='#XMMEA_EM && (PI in [200:12000]) && (PATTERN
in
> [0:12]) &&
> (FLAG==0)'
> edetect_chain imagesets=MOS1_image_full.fits eventsets=MOS1clean.fits
> attitudeset=1470_0500070601_AttHk.ds pimin=200 pimax=12000 ecf=2.0
> eboxl_list=MOS1_eboxlist_l.fits eboxm_list=MOS1_eboxlist_m.fits
> esp_nsplinenodes=16 eml_list=MOS1_emllist.fits esen_mlmin=15
> srcdisplay boxlistset=MOS1_emllist.fits imageset=MOS1_image_full.fits
> sourceradius=.01
> 
> #Sources removal
> region -w 0 -V 0 eventset=MOS1clean.fits operationstyle=global
> radiusstyle=userfixed fixedradius=35 srclisttab=MOS1_emllist.fits
> expression='ID_INST==0' outunit=detxy bkgregionset=MOS1reg.fits
>     evselect -w 0 -V 0 table=MOS1clean.fits
> expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1out.fits
> updateexposure=N writedss=Y withfilteredset=Y destruct=Y
keepfilteroutput=T
>       ftcopy "MOS1_emllist.fits[SRCLIST][#row <= 2 ]"
MOS1_emllist2.fits
> clobber=yes
>       region -w 0 -V 0 eventset=MOS1clean.fits operationstyle=global
> radiusstyle=userfixed fixedradius=300 srclisttab=MOS1_emllist2.fits
> expression='ID_INST==0' outunit=detxy bkgregionset=MOS1reg2.fits
>       evselect -w 0 -V 0 table=MOS1clean.fits
> expression="region(MOS1reg2.fits,DETX,DETY)" filteredset=MOS1out.fits
> updateexposure=N writedss=Y withfilteredset=Y destruct=Y
keepfilteroutput=T
>       evselect -w 0 -V 0 table=MOS1out.fits
> expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1cleanf.fits
> updateexposure=N writedss=Y withfilteredset=Y destruct=Y
keepfilteroutput=T
>   
> 
> #spectrum generation
> evselect table=MOS1cleanf.fits withspectrumset=y spectrumset=MOS1spec.fits
> energycolumn=PI spectralbinsize=5 withspecranges=y specchannelmin=0
> specchannelmax=11999 expression='#XMMEA_EM && (PATTERN<=12)
> && ((DETX,DETY) in
> circle (-50,-180,16000))' withfilteredset=y filteredset=MOS1cleanfs.fits
> backscale spectrumset=MOS1spec.fits badpixlocation=MOS1cleanfs.fits
> rmfgen spectrumset=MOS1spec.fits rmfset=MOS1.rmf
> arfgen spectrumset=MOS1spec.fits arfset=MOS1.arf withrmfset=y
rmfset=MOS1.rmf
> badpixlocation=MOS1cleanfs.fits detmaptype=flat extendedsource=y
> 
> #FWC file and spectrum generation
> evqpb table=1470_0500070601_EMOS1_S001_ImagingEvts.ds exposurefactor=2.0
> attfile=1470_0500070601_AttHk.ds outset=evqpb_MOS1.fits
> evselect table=evqpb_MOS1.fits withfilteredset=y
> filteredset=MOS1_qpb_clean.fits
> destruct=y keepfilteroutput=T expression="gti(MOS1gti.fits,TIME)"
>       evselect -w 0 -V 0 table=MOS1_qpb_clean.fits
> expression="region(MOS1reg.fits,DETX,DETY)" filteredset=MOS1out_fwc.fits
> updateexposure=N writedss=Y withfilteredset=Y destruct=Y
keepfilter

Message of length 6360 truncated


Reply 2

Resend
From: Pedro Rodriguez <xmmhelp@sciops.esa.int>
To: jjung11@alaska.edu
Subject: Re: evqpb spectrum shows more counts than science spectrum (PR#85684)
Date: Fri Sep 27 16:24:35 2019
Dear Jaewoong Jung,

current version of evqpb accumulates FWC data closest in time to the date of the
observation to analyze until it reaches an integration time longer than n times
the duration of the observation, where n is given in parameter exposurefactor .
The scaling of these FWC data is then as the ratio of the actual integration
time in the FWC and in the science. 

Due to long-term variability and scatter in the data we are aware that this
method might produce wrong background estimates for a given observation: if the
exposure factor is too high the long-term component might be smearead out but if
the exposure factor is too low the estimate might be wrong due to short-term
fluctuations and poor statistics. This seems to be your case, please, try a
larger exposurefactor (4-6) to check if the background estimate improves.

In any case, we are currently working on a different scaling method to try to
avoid the problems mentioned above.

Sincerely,

Pedro


Followup 1

Compose reply
Download message
From: Jaewoong Jung <jjung11@alaska.edu>
Date: Sat, 28 Sep 2019 16:53:10 -0800
Subject: Re: evqpb spectrum shows more counts than science spectrum (PR#85684)
To: Pedro Rodriguez <xmmhelp@sciops.esa.int>
--000000000000b99d380593a68b0e
Content-Type: text/plain; charset="UTF-8"

Dear Pedro,

I got better result for that event, when I used exposure factor 6.
However, for another event, like 0402380101, (2006-02-19) even when I used
exposure factor 6  I got negative count.
Code is same, except I changed observation/revolution number appropriately
and used

exposurefactor=6.0


Net count rate (cts/s) for Spectrum:1  -3.429e-02 +/- 9.238e-03 (-1.6 %
total)

 Spectral data counts: 59850


Should I increase exposure factor more?


Regards,

Jaewoong

On Fri, Sep 27, 2019 at 6:24 AM Pedro Rodriguez <xmmhelp@sciops.esa.int>
wrote:

> Dear Jaewoong Jung,
>
> current version of evqpb accumulates FWC data closest in time to the date
> of the
> observation to analyze until it reaches an integration time longer than n
> times
> the duration of the observation, where n is given in parameter
> exposurefactor .
> The scaling of these FWC data is then as the ratio of the actual
> integration
> time in the FWC and in the science.
>
> Due to long-term variability and scatter in the data we are aware that this
> method might produce wrong background estimates for a given observation:
> if the
> exposure factor is too high the long-term component might be smearead out
> but if
> the exposure factor is too low the estimate might be wrong due to
> short-term
> fluctuations and poor statistics. This seems to be your case, please, try a
> larger exposurefactor (4-6) to check if the background estimate improves.
>
> In any case, we are currently working on a different scaling method to try
> to
> avoid the problems mentioned above.
>
> Sincerely,
>
> Pedro
>
> This message is intended only for the recipient(s) named above. It may
> contain proprietary information and/or
> protected content. Any unauthorised disclosure, use, retention or
> dissemination is prohibited. If you have received
> this e-mail in error, please notify the sender immediately. ESA applies
> appropriate organisational measures to protect
> personal data, in case of data privacy queries, please contact the ESA
> Data Protection Officer (dpo@esa.int).
>
>
>

--000000000000b99d380593a68b0e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Dear=C2=A0Pedro,<div><br></div><div>I
got better result fo=
r that event, when I used exposure factor 6.</div><div>However, for
another=
 event, like 0402380101, (2006-02-19) even when I used exposure factor 6=C2=
=A0 I got negative count.</div><div>Code is same, except I changed
observat=
ion/revolution number appropriately and used=C2=A0<a name=3D"themesg" style=
=3D"color:rgb(0,0,0);font-family:Times;font-size:medium"><pre
style=3D"disp=
lay:inline!important">exposurefactor=3D6.0</pre></a></div><div>





<p class=3D"gmail-p1" style=3D"margin:0px;font-variant-numeric:normal;font-=
variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:no=
rmal;font-family:Menlo;color:rgb(0,0,0)"><span class=3D"gmail-s1"
style=3D"=
font-variant-ligatures:no-common-ligatures"><br></span></p><p
class=3D"gmai=
l-p1" style=3D"margin:0px;font-variant-numeric:normal;font-variant-east-asi=
an:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family=
:Menlo;color:rgb(0,0,0)"><span class=3D"gmail-s1"
style=3D"font-variant-lig=
atures:no-common-ligatures">Net count rate (cts/s) for Spectrum:1<span
clas=
s=3D"gmail-Apple-converted-space">=C2=A0 </span>-3.429e-02 +/-
9.238e-03 (-=
1.6 % total)</span></p>
<p class=3D"gmail-p1" style=3D"margin:0px;font-variant-numeric:normal;font-=
variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:no=
rmal;font-family:Menlo;color:rgb(0,0,0)"><span class=3D"gmail-s1"
style=3D"=
font-variant-ligatures:no-common-ligatures"><span
class=3D"gmail-Apple-conv=
erted-space">=C2=A0</span>Spectral data counts:
59850</span></p><p class=3D=
"gmail-p1" style=3D"margin:0px;font-variant-numeric:normal;font-variant-eas=
t-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-f=
amily:Menlo;color:rgb(0,0,0)"><span class=3D"gmail-s1"
style=3D"font-varian=
t-ligatures:no-common-ligatures"><br></span></p><p
class=3D"gmail-p1" style=
=3D"margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;f=
ont-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;colo=
r:rgb(0,0,0)"><span
style=3D"font-variant-ligatures:no-common-ligatures">Sh=
ould I increase exposure factor more?</span></p><p
class=3D"gmail-p1" style=
=3D"margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;f=
ont-stretch:normal;font-size:11px;line-hei

Message of length 8054 truncated


Reply 3

Resend
From: Pedro Rodriguez <xmmhelp@sciops.esa.int>
To: jjung11@alaska.edu
Subject: Re: evqpb spectrum shows more counts than science spectrum (PR#85684)
Date: Tue Oct  8 15:03:59 2019
Hello,

you may increase the exposure factor in evqpb or apply a correction factor to
the spectrum so that it matches the science spectrum in a given spectral region
if you do not expect contribution of astrophysical background, based on solid
grounds 

Pedro

> 
> Dear Pedro,
> 
> I got better result for that event, when I used exposure factor 6.
> However, for another event, like 0402380101, (2006-02-19) even when I used
> exposure factor 6  I got negative count.
> Code is same, except I changed observation/revolution number appropriately
> and used
> 
> exposurefactor=6.0
> 
> 
> Net count rate (cts/s) for Spectrum:1  -3.429e-02 +/- 9.238e-03 (-1.6 %
> total)
> 
>  Spectral data counts: 59850
> 
> 
> Should I increase exposure factor more?
> 
> 
> Regards,
> 
> Jaewoong
> 

Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page