Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest

Viewing Backgrounds/17960
Full headers

From: cocj@roe.ac.uk
Subject: Exposure correction of background files
Compose reply
Download message
Move To:
5 replies: 1 2 3 4 5
3 followups: 1 2 3

Private message: yes  no

Notes:

Notification:


Date: Fri, 27 May 2005 15:56:49 GMT
From: cocj@roe.ac.uk
To: xmmhelp@xmm.vilspa.esa.es
CC: cocj@roe.ac.uk
Subject: Exposure correction of background files
Full_Name: Olivia Johnson
Submission from: (NULL) (195.194.120.18)


Hi, 

I'm trying to use the background files made available by the Birmingham group in
a spectral analysis.  As they note, there are areas (such as at the aimpoint)
where there are deficits in counts left over from the source subtraction
process.  They state that this can be corrected using the provided exposure
maps, but I don't see how this is done for a spectral analysis.  Also, I don't
see any signs of vignetting in these exposure maps, though I think the event
lists themselves are vignetted?  I'd be most grateful if you would explain how
to account for the source-subtraction  artefacts in the background files (there
are similar artefacts in the Lumb event lists.)  Thanks.

Also, I think I've pieced together the process of using the background files as
set out by Lumb and Arnaud, but the specifics of this process are far from
clear.  Is there a standard set of analysis steps used by people in the know for
this, and could it be made public?  Considering the structure in the XMM
background, this must be a standard reduction process for properly-done cluster
analyses and getting it wrong will produce bad science results. 

Thanks again, 
-olivia





Reply 1

Resend
From: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
To: cocj@roe.ac.uk
Subject: Re: Exposure correction of background files (PR#17960)
Date: Mon May 30 09:29:07 2005
Dear Olivia, 

I have forwarded your question to the EPIC experts to have 
confirmation on that my draft answer was correct. 

I will come back to you as soon as I have an answer from them.
Many thanks for your patience.

With best regards

Maria


Maria Santos-Lleo, 
XMM-Newton SOC 
User Support Group


Reply 2

Resend
From: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
To: cocj@roe.ac.uk
Subject: Re: Exposure correction of background files (PR#17960)
Date: Thu Jun  2 12:11:49 2005
Dear Olivia, 

Sorry for the delay in coming back to you, but as I told you, I wanted 
to cross-check with the EPIC experts before sending you my answer to
your questions: 

>> Hi,
>> I'm trying to use the background files made available by the Birmingham
group
in
>> a spectral analysis.  As they note, there are areas (such as at the
aimpoint)
>> where there are deficits in counts left over from the source
subtraction
>> process.  They state that this can be corrected using the provided
exposure
>> maps, but I don't see how this is done for a spectral analysis. 
>

for spectral analysis you do not need to use the provided exposure maps, since
what you need is to extract a background spectrum and you can do it
following the standard procedures. As a result of the effect mentioned
above, the background spectrum may have lower signal to noise than a
background spectrum extracted from a different region, but this will not
mean that the background spectrum is incorrect.


>>  Also, I don't
>> see any signs of vignetting in these exposure maps, though I think the
event
>> lists themselves are vignetted?  I'd be most grateful if you would
explain
how
>> to account for the source-subtraction  artefacts in the background
files
(there
>> are similar artefacts in the Lumb event lists.)  Thanks.
>
>

It is important to note here that the following two items need to 
be considered separately: 

1) vignetting: yes, events are vignetted due to the effective area of the
telescope but this is taken into account in the arf file. Therefore, 
for spectral analysis, you can correct for vignetting either by using 
the evigweight sas task and then using on-axis matrices or by computing 
different rmf and arf for each extracted source and its corresponding 
background. Please consult the help on evigweight for more information 
about what it is done by this task.

2) source-subtraction artefacts: well yes, there are artefacts but 
these background files are the best blank sky fields that we 
currently can offer.

>>
>> Also, I think I've pieced together the process of using the background
files
as
>> set out by Lumb and Arnaud, but the specifics of this process are far
from
>> clear.  Is there a standard set of analysis steps used by people in the
know
for
>> this, and could it be made public?  Considering the structure in the
XMM
>> background, this must be a standard reduction process for properly-done
cluster
>> analyses and getting it wrong will produce bad science results.


The best advice to date that you can find in our web is the following
SAS Workshop presentation: 'Analysis of extended sources with EPIC' 

http://xmm.vilspa.esa.es/sas/workshops/VILSPA_0210/

this presentation will be updated very soon, in a couple of weeks, since
the next SAS Workshop will be held next week. The new version does not
contain many changes, so I do not think is worth to wait for it. 


Please note that clarifying the EPIC background treatment as well as
devolping new tools for this purpose is in the XMM-Newton SOC 
task list with high priority. This point is also in the list of 
recommendations from the XMM-Newton Users Group (UG). 
However, last year, the UG only gave it a second priority,
after the cross-calibration. It is only this year when it has got 
the same priority as two other pending calibration topics. 
Therefore, we have now put it in our top priority list, but we need 
to share the same resources for all three topics. Hence, the time
scale for these threads and instructions to be public will still be 
of the order of months. We anyway appreciate your comments and will 
try to take them into account and acknowledge your patience. 

Hope all this helps

With best regards

Maria





Followup 1

Compose reply
Download message
Date: Fri, 03 Jun 2005 10:18:33 +0100 (BST)
From: Olivia Johnson <cocj@roe.ac.uk>
Subject: Re: Exposure correction of background files (PR#17960)
To: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
Hi Maria,

  Thanks very much for your reply.

-Olivia




Followup 2

Compose reply
Download message
Date: Fri, 03 Jun 2005 11:15:31 +0100 (BST)
From: Olivia Johnson <cocj@roe.ac.uk>
Subject: Re: Exposure correction of background files (PR#17960)
To: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
Hi again Maria,

  Sorry, I thanked you a wee bit too soon.  Unfortunately, having read
through the presentation you suggested it is still unclear to me how to
proceed.  There are two specific bits that are unclear:

  First, the papers I've read on background subtraction (Arnaud, Lumb,
Ponman) all discuss two separate corrections to the blank field background
data, a multiplicative factor to account for variation in the internal
background and an additive factor to account for difference in the cosmic
X-ray background.  The Guainazzi presentation applys only a single
multiplicative correction to the weights of each background count based on
the hard-band light curves.  Question 1 is: Doesn't this neglect the
significant cosmic background variance?

  (In my own data, I've done the non-X-ray background scaling and
subtracted and am left with basically no source counts, despite there
being a visible excess of cluster emission - about 900 counts over the 3
detectors - and I assume this is because the cosmic background in the
background fields was higher than in my observation.  I am therefore
stalled trying to work out how to subtract this component away using
off-source spectral differences since the statistics per bin are quite
poor in the observation data.  A corellary question, I suppose, is whether
my emission is likely to be just too faint to get a good spectrum from
with the currently available background subtraction methods.)

  My second problem is with the use of evigweight.  I had decided to avoid
this because it apparently degrades the statistics of the data and, as
I've said, the emission I am studying is faint.  Furthermore, the
description of this task says it is for use with inhomogenous sources (I
assume this is basically because there is no SAS equivalent to the CIAO
task which produces weighted arfs?) but further on says there is no
correction for chip gaps, etc., and the best way to get around this is to
assume your source is homogenous!  So question 2.1 is: Is there a reason
why you need to alter the WEIGHT column of the background event list
rather than scaling the extracted spectrum?

  Question 2.2 is: If I have to use evigweight and the central .arf, is it
correct to use the central .rmf or the .rmf of the source region?
Guainazzi and Arnaud both say to use the central .arf but the evigweight
description says "Of course the response matrix should be taken (via
rmfgen) in the true detector region (not at the center)".  This makes
sense to me, since the redistribution is position-dependent and would not
have been corrected by evigweight.

  Thanks again, Maria, and sorry to give you more to do.  I appreciate how
quick and useful your replies always are.  I worked in Chandra calibration
for many years and realize that the folks manning the helpdesk are only
the front line of an extremely large and complex effort.

Cheers,
-Olivia



On Thu, 2 Jun 2005, Maria Santos-Lleo wrote:

> Dear Olivia,
>
> Sorry for the delay in coming back to you, but as I told you, I wanted
> to cross-check with the EPIC experts before sending you my answer to
> your questions:
>
> >> Hi,
> >> I'm trying to use the background files made available by the
Birmingham group
> in
> >> a spectral analysis.  As they note, there are areas (such as at
the
> aimpoint)
> >> where there are deficits in counts left over from the source
subtraction
> >> process.  They state that this can be corrected using the provided
exposure
> >> maps, but I don't see how this is done for a spectral analysis.
> >
>
> for spectral analysis you do not need to use the provided exposure maps,
since
> what you need is to extract a background spectrum and you can do it
> following the standard procedures. As a result of the effect mentioned
> above, the background spectrum may have lower signal to noise than a
> background spectrum extracted from a different region, but this will not
> mean that the background spectrum is incorrect.
>
>
> >>  Also, I don't
> >> see any signs of vignetting in these exposure maps, though I think
the event
> >> lists themselves are vignetted?  I'd be most grateful if you would
explain
> how
> >> to account for the source-subtraction  artefacts in the background
files
> (there
> >> are similar artefacts in the Lumb event lists.)  Thanks.
> >
> >
>
> It is important to note here that the following two items need to
> be considered separately:
>
> 1) vignetting: yes, events are vignetted due to the effective area of the
> telescope but this is taken into account in the arf file. Therefore,
> for spectral analysis, you can correct for vignetting either by using
> the evigweight sas task and then using on-axis matrices or by computing
> different rmf and arf for each extracted source and its corresponding
> background.

Message of length 7224 truncated


Reply 3

Resend
From: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
To: cocj@roe.ac.uk
Subject: Re: Exposure correction of background files (PR#17960)
Date: Fri Jun  3 14:49:12 2005
Hi Olivia, 

I have forwarded your question again to the EPIC experts. 
However, they cannot go into such kind of details now.
They are holding a background workshop in a few weeks. 
Before this point, we do not even have a clear idea
on the time scale for significant progress to be done. 
See also further comments on this below. 

>   First, the papers I've read on background subtraction (Arnaud, Lumb,
> Ponman) all discuss two separate corrections to the blank field background
> data, a multiplicative factor to account for variation in the internal
> background and an additive factor to account for difference in the cosmic
> X-ray background.  The Guainazzi presentation applys only a single
> multiplicative correction to the weights of each background count based on
> the hard-band light curves.  Question 1 is: Doesn't this neglect the
> significant cosmic background variance?
> 
>   (In my own data, I've done the non-X-ray background scaling and
> subtracted and am left with basically no source counts, despite there
> being a visible excess of cluster emission - about 900 counts over the 3
> detectors - and I assume this is because the cosmic background in the
> background fields was higher than in my observation.  I am therefore
> stalled trying to work out how to subtract this component away using
> off-source spectral differences since the statistics per bin are quite
> poor in the observation data.  A corellary question, I suppose, is whether
> my emission is likely to be just too faint to get a good spectrum from
> with the currently available background subtraction methods.)
> 

I am afraid those questions go too much in the scientific analysis
of the data, for which we cannot provide such a detailed help. We
would need to properly analyse the data to asses whether there is 
indeed cluster signal above background. I think this falls more on
the science analysis than on the SAS processing of the data. 

As I told you before, the SOC is right now investing a large effort
to help users to deal with background analysis, especially for 
extended sources. However, the EPIC team cannot at this moment 
provide more information than what is on the web. 

Our intention is to provide some instructions through a dedicated
web page. You can keep up checking our web, but I cannot promise
any real progress before a few months in the very minimum. I guess
that if a significant progress is made, we will also advertise it 
in one XMM-News electronic circular. 

>   My second problem is with the use of evigweight.  I had decided to avoid
> this because it apparently degrades the statistics of the data and, as
> I've said, the emission I am studying is faint.  Furthermore, the
> description of this task says it is for use with inhomogenous sources (I
> assume this is basically because there is no SAS equivalent to the CIAO
> task which produces weighted arfs?) but further on says there is no
> correction for chip gaps, etc., and the best way to get around this is to
> assume your source is homogenous!  So question 2.1 is: Is there a reason
> why you need to alter the WEIGHT column of the background event list
> rather than scaling the extracted spectrum?
> 
>   Question 2.2 is: If I have to use evigweight and the central .arf, is it
> correct to use the central .rmf or the .rmf of the source region?
> Guainazzi and Arnaud both say to use the central .arf but the evigweight
> description says "Of course the response matrix should be taken (via
> rmfgen) in the true detector region (not at the center)".  This makes
> sense to me, since the redistribution is position-dependent and would not
> have been corrected by evigweight.
> 


I think you should simply forget about evigweight. It was suggested as
an alternative to the best method which is to extract the spectrum and 
then generate the arf and rmf for it. I beleive evigweight is only 
useful for people that want to extract spectra from many sources 
and want to avoid generating a lot of matrices: one rmf and arf for 
each of these sources. Therefore it was probably my fault to suggest 
its use and create some more confusion. 

Hope this helps

With best regards

Maria



Followup 3

Compose reply
Download message
Date: Mon, 06 Jun 2005 12:06:15 +0100 (BST)
From: Olivia Johnson <cocj@roe.ac.uk>
Subject: Re: Exposure correction of background files (PR#17960)
To: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
Hi Maria,

  Thanks for your reply.  At this point, I will have to abandon my attempt
to get a background-subtracted spectrum for my faint cluster emission,
though the recent paper by Lumb suggests that this should be possible.
I'll check back on your webpages from time to time to see if there's been
any progress on the background issue.

  Perhaps you could take forward to discussions on background calibration
the fact that the existence of two kinds of background emission requiring
different corrections is a factor which appears to limit the extraction of
science information from faint emission in some cases.  Also, while the
analysis of faint emission in my dataset is not, as you say, in your
remit, it would be great to have some indication of the level at which the
background can be successfully subtracted using standard methods delivered
along with the background tools to save users from wasting time analysing
signal which doesn't exceed the quality of the background calibration.
Finally, while I understand that you cannot at this time offer better
advice than what's on your webpages, in the case of the evigweight, the
information on two of your webpages is contradictory and it would be good
to know who to believe.

Thanks again for your time,
-olivia



>
> I have forwarded your question again to the EPIC experts.
> However, they cannot go into such kind of details now.
> They are holding a background workshop in a few weeks.
> Before this point, we do not even have a clear idea
> on the time scale for significant progress to be done.
> See also further comments on this below.
>
> >   First, the papers I've read on background subtraction (Arnaud, Lumb,
> > Ponman) all discuss two separate corrections to the blank field
background
> > data, a multiplicative factor to account for variation in the internal
> > background and an additive factor to account for difference in the
cosmic
> > X-ray background.  The Guainazzi presentation applys only a single
> > multiplicative correction to the weights of each background count
based on
> > the hard-band light curves.  Question 1 is: Doesn't this neglect the
> > significant cosmic background variance?
> >
> >   (In my own data, I've done the non-X-ray background scaling and
> > subtracted and am left with basically no source counts, despite there
> > being a visible excess of cluster emission - about 900 counts over the
3
> > detectors - and I assume this is because the cosmic background in the
> > background fields was higher than in my observation.  I am therefore
> > stalled trying to work out how to subtract this component away using
> > off-source spectral differences since the statistics per bin are quite
> > poor in the observation data.  A corellary question, I suppose, is
whether
> > my emission is likely to be just too faint to get a good spectrum from
> > with the currently available background subtraction methods.)
> >
>
> I am afraid those questions go too much in the scientific analysis
> of the data, for which we cannot provide such a detailed help. We
> would need to properly analyse the data to asses whether there is
> indeed cluster signal above background. I think this falls more on
> the science analysis than on the SAS processing of the data.
>
> As I told you before, the SOC is right now investing a large effort
> to help users to deal with background analysis, especially for
> extended sources. However, the EPIC team cannot at this moment
> provide more information than what is on the web.
>
> Our intention is to provide some instructions through a dedicated
> web page. You can keep up checking our web, but I cannot promise
> any real progress before a few months in the very minimum. I guess
> that if a significant progress is made, we will also advertise it
> in one XMM-News electronic circular.
>
> >   My second problem is with the use of evigweight.  I had decided to
avoid
> > this because it apparently degrades the statistics of the data and, as
> > I've said, the emission I am studying is faint.  Furthermore, the
> > description of this task says it is for use with inhomogenous sources
(I
> > assume this is basically because there is no SAS equivalent to the
CIAO
> > task which produces weighted arfs?) but further on says there is no
> > correction for chip gaps, etc., and the best way to get around this is
to
> > assume your source is homogenous!  So question 2.1 is: Is there a
reason
> > why you need to alter the WEIGHT column of the background event list
> > rather than scaling the extracted spectrum?
> >
> >   Question 2.2 is: If I have to use evigweight and the central .arf,
is it
> > correct to use the central .rmf or the .rmf of the source region?
> &g

Message of length 5956 truncated


Reply 4

Resend
From: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
To: cocj@roe.ac.uk
Subject: Re: Exposure correction of background files (PR#17960)
Date: Tue Jun  7 11:46:01 2005
Dear Olivia, 

I am very sorry that you think you are now stuck in your work.

I have forwarded again your comments to the EPIC experts. As I 
told you before, we are taking into account all your comments 
about EPIC background subtraction, included those sent in 
previous related questions (like source and background extraction 
with complicated regions due to source extent). As mentioned 
before, we appreciate these comments since they are useful to 
better understand what are the needs in this area. We have 
given the highest priority to the EPIC background analysis. 

The problem you mention about the evigweight documentation 
has been forwarded to the SAS team, they are trying 
to solve it. I did not mention this fact before because you 
told me you did not want to use evigweight and I had agreed
with you that it is better. Now I understand that you would
rather like to be informed. I will therefore let you know as 
soon as the SAS experts reply to my question. 

I will let you know about anything the EPIC experts can tell
me. Sorry for not being able to provide you better answers 
right now, but, again, I can asure you that the background 
treatment has a very high priority in our task list and that 
we will take into accout your comments, which are always 
welcome.

With best wishes

Maria



Reply 5

Resend
From: Maria Santos-Lleo <xmmhelp@xmm.vilspa.esa.es>
To: cocj@roe.ac.uk
Subject: Re: Exposure correction of background files (PR#17960)
Date: Fri Jul  8 14:48:42 2005
Hi Olivia, 

I am afraid I cannot give you further news related to background 
analysis, nor further help. However, I wanted to let you know 
that a few days ago the following information was added in our
web pages:

------------------------------------------------------------

Helsinki University Observatory

Supporting data for astro-ph/0504362 "XMM-Newton EPIC background 
modeling for extended sources" (ApJ accepted), J. Nevalainen, 
M. Markevitch and D. Lumb available at:
http://www.astro.helsinki.fi/~jnevalai/XMMbkg/.
-------------------------------------------------------------

please, keep on looking into our web pages for further 
information that may go into there in the coming months. 

With kind regards

Maria


Up to top level
AO15   AO16   AO17   AO18   AO19   Backgrounds   Calibration   Conference   Data   Docs   EPICMOS   EPICpn   Feedback   Gallery   Misc   OM   Pending   PhD_Theses   Publications   RGS   RadMonitor   SAS_Hardware   SAS_WS   SASv16.0   SASv16.0_Installation   SASv16.1   SASv16.1_Installation   SASv17.0   SASv17.0_Installation   SASv18.0   SASv18.0_Installation   SciSim   Simulators_other   Suggestions   Trash   Visibility   XMM-bouncing   XMM-news   XRPS   XSA   esas   incoming  

Logged in as guest


Please make your (short) question the subject of your request!


Web interface using JitterBug ... back to the XMM home page