Identifying and acquiring relevant X-ray observations

This tutorial will explain the basic concepts and components of the X-ray astronomy Python module ‘Democratising Archival X-ray Astronomy’ (DAXA). We will particularly focus on the various ‘mission’ classes (implemented for each of the X-ray telescopes that DAXA supports), and the functionality that allows for large numbers of observations to be selected and downloaded.

DAXA mission classes allow the user to interact with and search various X-ray telescope archives, all through an identical interface, within a single module, and through Python commands. It should be simple for anyone with a passing familiarity with Python to identify and acquire X-ray data relevant to their research.

Import Statements

[1]:
from daxa.mission import MISS_INDEX, XMMPointed, Chandra, eRASS1DE, eROSITACalPV, NuSTARPointed, Swift, \
    ROSATAllSky, ROSATPointed, ASCA, INTEGRALPointed, Suzaku

from datetime import date
import numpy as np
from astropy.units import Quantity

What missions are available?

We have implemented support for access and searching the archives of many X-ray telescopes; we would also be willing to provide an interface to the data archives of other X-ray telescopes, if feasible - please feel free to reach out using the support page if you think there is one we should add.

Some telescopes (such as the ROSATPointed/ROSATAllSky and eROSITACalPV/eRASS1DE classes) are not uniquely represented by a single DAXA mission - this is generally the case when the telescope in question has been used in distinctly different ways (e.g. ROSAT had a survey phase and a pointed phase), such that the data for one mode may not be relevant to all applications.

[2]:
MISS_INDEX
[2]:
{'xmm_pointed': daxa.mission.xmm.XMMPointed,
 'nustar_pointed': daxa.mission.nustar.NuSTARPointed,
 'erosita_calpv': daxa.mission.erosita.eROSITACalPV,
 'erosita_all_sky_de_dr1': daxa.mission.erosita.eRASS1DE,
 'chandra': daxa.mission.chandra.Chandra,
 'rosat_all_sky': daxa.mission.rosat.ROSATAllSky,
 'rosat_pointed': daxa.mission.rosat.ROSATPointed,
 'swift': daxa.mission.swift.Swift,
 'suzaku': daxa.mission.suzaku.Suzaku,
 'asca': daxa.mission.asca.ASCA,
 'integral_pointed': daxa.mission.integral.INTEGRALPointed}

XMM-Pointed

This class is for acquiring XMM-Newton data, particularly that taken when the telescope is pointing at specific targets; i.e. it cannot be used to filter and download data taken when the telescope is slewing from one target to the next. XMM is still in use, so the number of observations that are available are constantly increasing - this also means that some data are still in a proprietary period, and DAXA will not be able to access them. Note that only raw, unprocessed, data can currently be downloaded for XMM using DAXA.

The three EPIC instruments are supported (PN, MOS1, and MOS2), as well as the two grating spectrometers (RGS1 and RGS2). We also support the acquisition of XMM optical monitor (OM) data, but support for processing it is more limited.

Values that can be passed to the insts argument of XMMPointed on declaration (either as single strings or as part of a list):

  • PN

  • MOS1

  • MOS2

  • RGS1

  • RGS2

  • OM

[3]:
xm = XMMPointed()
/Users/dt237/code/DAXA/daxa/mission/xmm.py:83: UserWarning: 140 of the 17701 observations located for this mission have been removed due to NaN RA or Dec values
  self._fetch_obs_info()

Chandra

The class for acquiring Chandra data, specifically taken when pointing at a target - Chandra does not have easily accessible ‘slew’ data, and this class does not consider it at all.

Data from all instruments can be downloaded by DAXA - though unlike XMM there are not data being taken simultaneously, so each ObsID is generally only associated with one instrument. Note though that the gratings (HETG and LETG) are used with another instrument, so cannot be passed by themselves.

Note that the standard format of Chandra data can be downloaded using DAXA, allowing for the use of the standard Chandra re-processing scripts. This standard download includes pre-made images.

Values that can be passed to the insts argument of Chandra (either as single string or as part of a list):

  • ACIS-I

  • ACIS-S

  • HRC-I

  • HRC-S

  • LETG [an instrument that this grating was used with must be specified]

  • HETG [an instrument that this grating was used with must be specified]

[4]:
ch = Chandra()

eROSITA All-Sky DR1 (German Half)

The class for acquiring data from the first data release by the German part of the eROSITA consortium - this covers half the sky to 1/8th the planned final survey depth of eROSITA. All of this data is taken in survey mode, where the telescope is constantly slewing. Note that the data downloads can include pre-generated images and exposure maps.

The eROSITA telescope is made up of 7 telescope modules that observe simultaneously, and which can be individually selected on declaration with the insts argument - it is not recommended to use DAXA to limit the telescope modules being considered, as we crudely modify the event lists to remove events from non-selected telescope modules.

Values that can be passed to the insts argument of eRASS1DE (either as a single string or as part of a list):

  • TM1

  • TM2

  • TM3

  • TM4

  • TM5

  • TM6

  • TM7

[5]:
ea = eRASS1DE()

eROSITA Calibration and Performance Verification

This class provides DAXA access to the data from the calibration and performance verification stages of the eROSITA mission, including all pointed observations and survey regions (such as the eROSITA Final Equatorial-Depth Survey; eFEDS). Note that only event list data can be downloaded for this class.

The eROSITA telescope is made up of 7 telescope modules that observe simultaneously, and which can be individually selected on declaration with the insts argument - it is not recommended to use DAXA to limit the telescope modules being considered, as we crudely modify the event lists to remove events from non-selected telescope modules.

Values that can be passed to the insts argument of eROSITACalPV (either as a single string or as part of a list):

  • TM1

  • TM2

  • TM3

  • TM4

  • TM5

  • TM6

  • TM7

[6]:
ecpv = eROSITACalPV()

NuSTAR-Pointed

This class is for acquiring NuSTAR data, particularly that taken when the telescope is pointing at specific targets; i.e. it cannot be used to filter and download data taken when the telescope is slewing from one target to the next. NuSTAR is still in use, so the number of observations that are available are constantly increasing - this also means that some data are still in a proprietary period, and DAXA will not be able to access them. Note that data downloads can optionally include pre-generated images

NuSTAR has two instruments that observe simultaneously, and are essentially identical, Focal Plane Module A & B.

Values that can be passed to the insts argument of NuSTARPointed on declaration (either as single strings or as part of a list):

  • FPMA

  • FPMB

[7]:
nu = NuSTARPointed()

Swift

This class is for acquiring Swift data. Swift is still in use, so the number of observations that are available are constantly increasing. Also, due to the Swift’s primary mission of rapid transient follow-up, and how observations are split up, the table of available observations is unusually large (only INTEGRAL is comparable) - as such this class may take several minutes to declare, depending on your internet connection.

Swift has three instruments that generally observe simultaneously; the X-ray Telescope (XRT), the Burst Alert Telescope (BAT) which is has poor spatial resolution but has a large field of view and is sensitive to very high energy photons, and the Ultraviolet and Optical Telescope (UVOT). Note that data downloads can optionally include pre-generated images, but not for BAT observations.

Values that can be passed to the insts argument of Swift on declaration (either as single strings or as part of a list); the XRT and BAT instruments are selected by default:

  • XRT

  • BAT

  • UVOT

[8]:
sw = Swift()
/Users/dt237/code/DAXA/daxa/mission/swift.py:122: UserWarning: 599 of the 354020 observations located for Swift have been removed due to all instrument exposures being zero.
  self._fetch_obs_info()
/Users/dt237/code/DAXA/daxa/mission/swift.py:122: UserWarning: 17 of the 354020 observations located for Swift have been removed due to all chosen instrument (XRT, BAT) exposures being zero.
  self._fetch_obs_info()

ROSAT All-Sky Survey

This class provides access to data taken by ROSAT during its all-sky survey. Though ROSAT had multiple instruments, this was all taken with the position-sensitive proportional counters (PSPC) - specifically with the ‘PSPC-C’ instrument (ROSAT had two PSPC instruments). The initial all-sky survey was abandoned after an accidental pass over the sun destroyed PSPC-C, but follow-up observations with PSPC-B were taken towards the end of ROSAT’s lifetime to complete the survey (these are not included in this class).

Note: Data acquired through this class will include just event lists by default, but can also include pre-generated images and exposure maps.

[9]:
ra = ROSATAllSky()

ROSAT-Pointed

This class provides access to data taken by ROSAT during the pointed phase of its lifetime (including follow-up observations used to complete the all-sky survey). ROSAT instruments could not observe simultaneously, so each separate observation uses a single instrument. Note: Data acquired through this class will include just event lists by default, but can also include pre-generated images (PSPC & HRI) and exposure maps (just PSPC).

Values that can be passed to the insts argument of ROSATPointed on declaration (either as single strings or as part of a list):

  • PSPCB

  • PSPCC

  • HRI

[10]:
rp = ROSATPointed()

Suzaku

This class provides access to data taken by the Suzaku X-ray telescope during pointed observations (data taken while slewing are not included in the public archive). Note: Data acquired through this class will include just event lists by default, but can also include pre-generated images.

We provide access to XIS data, but not XRS (as the cooling system was damaged soon after launch) or HXD (as it was not an imaging instrument).

Values that can be passed to the insts argument of Suzaku on declaration (either as single strings or as part of a list):

  • XIS0

  • XIS1

  • XIS2

  • XIS3

[11]:
su = Suzaku()
/Users/dt237/code/DAXA/daxa/mission/suzaku.py:109: UserWarning: 14 of the 3055 observations located for Suzaku have been removed due to all instrument exposures being zero.
  self._fetch_obs_info()

ASCA

This class provides access to data taken by the ASCA X-ray telescope during pointed observations (we cannot find anywhere to access the data taken whilst slewing). Note: Data acquired through this class will include just event lists by default, but can also include pre-generated images, spectra, and lightcurves.

Values that can be passed to the insts argument of ASCA on declaration (either as single strings or as part of a list):

  • SIS0

  • SIS1

  • GIS2

  • GIS3

[12]:
asca = ASCA()
/Users/dt237/code/DAXA/daxa/mission/asca.py:125: UserWarning: 5 of the 3079 observations located for ASCA have been removed due to all instrument exposures being zero.
  self._fetch_obs_info()

INTEGRAL-Pointed

This class is for acquiring INTEGRAL data. INTEGRAL is still in use, so the number of observations that are available are still increasing (though operations will see around the end of 2024). The table of available observations is unusually large (only Swift is comparable) - as such this class may take several minutes to declare, depending on your internet connection.

INTEGRAl has a selection of instruments that cover different parts of the X-ray and Gamma-ray energy range - most of them are based on the ‘coded mask’ technology, and so have very limited spatial resolution. Note that only raw data/calibration files can be downloaded through DAXA, there are no pre-processed images available.

Values that can be passed to the insts argument of INTEGRALPointed on declaration (either as single strings or as part of a list):

  • JEMX1

  • JEMX2

  • ISGRI

  • PICsIT

  • SPI

[13]:
inte = INTEGRALPointed()
/Users/dt237/code/DAXA/daxa/mission/integral.py:110: UserWarning: 241 of the 205930 observations located for INTEGRAL have been removed due to all instrument exposures being zero.
  self._fetch_obs_info()
/Users/dt237/code/DAXA/daxa/mission/integral.py:110: UserWarning: 7403 of the 205930 observations located for INTEGRAL have been removed due to all chosen instrument (JEMX1, JEMX2, ISGRI) exposures being zero.
  self._fetch_obs_info()

Mission properties

Here we run through the basic properties that each of the DAXA mission classes share. We also show examples, particularly in cases where differences between telescopes result in us assigning different values for particular properties.

Name

The name assigned to each mission class, so that they can be differentiated both by the user and by DAXA functions. Each mission class has two names, the ‘internal DAXA name’ (used by DAXA to identify missions) and the ‘pretty name’, which is typically in a more aethsetically pleasing format.

For instance, we show the ‘name’ and ‘pretty name’ of the XMMPointed and eROSITA All-Sky Survey 1 classes:

[14]:
print(xm.name)
print(xm.pretty_name, '\n')

print(ea.name)
print(ea.pretty_name)
xmm_pointed
XMM-Newton Pointed

erosita_all_sky_de_dr1
eRASS DE:1

All Instruments & Chosen Instruments

Most telescopes have multiple instruments, though not all are necessarily selected by default. This can be for a number of reasons, but is generally because they either aren’t suited to archival/serindipitious science (which is the primary reason this module exists) or because they aren’t X-ray telescopes (like the optical monitors on XMM and Swift).

The instruments whose data is to be acquired are generally specified when the mission class is declared (using the insts argument), but can also be set through the chosen_instruments property.

Every available instrument for a mission is stored in the all_mission_instruments property:

[15]:
print(rp.all_mission_instruments)
print(ch.all_mission_instruments)
['PSPCB', 'PSPCC', 'HRI']
['ACIS-I', 'ACIS-S', 'HRC-I', 'HRC-S', 'HETG', 'LETG']

The selected instruments (normally specified on declaration) are stored in the chosen_instruments property, which can also be set:

[16]:
print(rp.chosen_instruments)
print(ch.chosen_instruments)
ch.chosen_instruments = ['ACIS-I', 'ACIS-S']
print(ch.chosen_instruments)
['PSPCB', 'PSPCC', 'HRI']
['ACIS-I', 'ACIS-S', 'HRC-I', 'HRC-S']
['ACIS-I', 'ACIS-S']

ObsID Regular Expression

Each of the mission’s observations are uniquely identified by an ‘ObsID’, and each telescope/mission has a different format of ObsID (generally just made up of numeric characters) - there are points where the mission class may have to check the format of a supplied ObsID, and it does that by comparing to the ObsID regular expression:

[17]:
print(xm.id_regex)
print(rp.id_regex)
^[0-9]{10}$
^(RH|rh|RP|rp|RF|rf|WH|wh|WP|wp|WF|wf)\d{6}([A-Z]\d{2}|)$

Field of View

The field of view (FoV) values attached to DAXA mission classes represent the half-width (or radius) of the region of sky that a telescope/instrument observes. Given that each telescope instrument tends to have a unique geometry, this is a simplification, but it is beneficial to have a single number that represents how much of the sky an instrument can see.

In the simplest cases, the FoV property is just a single quantity, meaning that there is either only one instrument, or that every instrument has the same field of view. In other cases there may be multiple instruments associated with a mission, in which case they will all have their own entry in a FoV dictionary.

Finally, some telescopes (such as Chandra) have instruments which have irregular geometries (the ACIS-S and HRC-S chips), or that are frequently used in different observational modes that enable and disable different parts of the sensor. As such the actual coverage of the FoV can vary dramatically, in cases like these we will have gone with the half-width of the longest possible side of the field of view.

[18]:
print(xm.pretty_name, '-', xm.fov, '\n')
print(rp.pretty_name, '-', rp.fov, '\n')
print(sw.pretty_name, '-', sw.fov, '\n')
print(ch.pretty_name, '-', ch.fov)
XMM-Newton Pointed - 15.0 arcmin

ROSAT Pointed - {'PSPCB': <Quantity 60. arcmin>, 'PSPCC': <Quantity 60. arcmin>, 'HRI': <Quantity 19. arcmin>}

Swift - {'XRT': <Quantity 11.8 arcmin>, 'BAT': <Quantity 50. arcmin>, 'UVOT': <Quantity 8.5 arcmin>}

Chandra - {'ACIS-I': <Quantity 27.8 arcmin>, 'ACIS-S': <Quantity 27.8 arcmin>, 'HRC-I': <Quantity 49.5 arcmin>, 'HRC-S': <Quantity 49.5 arcmin>}
/var/folders/td/gw9qkx6d3szb1nkt_cfvcbzm000vzl/T/ipykernel_23287/579883568.py:4: UserWarning: Chandra FoV are difficult to define, as they can be strongly dependant on observation mode; as such take these as very approximate.
  print(ch.pretty_name, '-', ch.fov)

Coordinate Frame

This property contains the coordinate frame for the central positions of observations taken by the mission - this is largely irrelevant to the user, and will be used in cases where the mission class needs to compare an input coordinate to an observation coordinate.

Also, practically speaking the difference between the ICRS and FK5 frames (most commonly used) is neglible compared to the typical spatial uncertainty involved in X-ray data:

[19]:
print(ch.coord_frame)
print(asca.coord_frame)
<class 'astropy.coordinates.builtin_frames.icrs.ICRS'>
<class 'astropy.coordinates.builtin_frames.fk5.FK5'>

Pre-processed Energy Bands

This property will contain the upper and lower energy bounds available for pre-processed data products for a particular mission (if their online dataset supplies energy-bound data products) - these energy bounds are provided on an instrument level (as some missions provide different energy-bound products for different instruments). The left hand column indicates the lower energy bound, and the right hand column the upper energy bound.

An energy bound being present here does not guarantee that all products supplied by the mission online dataset are available in that bound - e.g. some missions provide bound images and a single, general, exposure map.

[20]:
rp.preprocessed_energy_bands
[20]:
{'PSPCB': <Quantity [[0.07, 2.4 ],
            [0.07, 0.4 ],
            [0.4 , 2.4 ]] keV>,
 'PSPCC': <Quantity [[0.07, 2.4 ],
            [0.07, 0.4 ],
            [0.4 , 2.4 ]] keV>,
 'HRI': <Quantity [[0.07, 2.4 ]] keV>}

If the mission in question cannot provide pre-processed data products that are energy bound, then an error will be raised:

[21]:
inte.preprocessed_energy_bands
---------------------------------------------------------------------------
PreProcessedNotSupportedError             Traceback (most recent call last)
Cell In [21], line 1
----> 1 inte.preprocessed_energy_bands

File ~/code/DAXA/daxa/mission/base.py:730, in BaseMission.preprocessed_energy_bands(self)
    727 # If this attribute is not set then we're going to assume that the archive doesn't provide any products
    728 #  which are energy bound
    729 if self._template_en_trans is None:
--> 730     raise PreProcessedNotSupportedError("This mission's archive does not supply pre-processed products within "
    731                                         "specific energy bands.")
    733 # The attribute is organized as a nested dictionary - with two possible configurations, one with instrument
    734 #  names as top level keys, then lower level keys being lower energy bounds, and the
    735 #  lowest level keys being upper energy bounds - the other configuration is the same, but doesn't have top
    736 #  level instrument keys (these then apply to all instruments of a mission).
    737 if isinstance(list(self._template_en_trans.keys())[0], Quantity):

PreProcessedNotSupportedError: This mission's archive does not supply pre-processed products within specific energy bands.

Observation Information

One of the most important properties of a DAXA mission class - this returns a dataframe of all the observations that are associated with a mission. This can include observations that are not yet publicly available (i.e. they are still in a proprietary period), but will never include observations that are planned but haven’t been taken yet.

In most cases this data is dynamically updated when a mission is declared (i.e. the table is pulled down from a mission server) - this is not the case for eROSITACalPV and eRASS1DE. Please also note that the Swift and INTEGRALPointed missions have very large ``all_obs_info`` tables due to the way their data/observations are organised.

Some information will be constant across telescopes, and some will be mission specific. We present truncated versions of all_obs_info for every DAXA mission as of the current date:

[22]:
date.today().strftime('%d-%B-%Y')
[22]:
'24-April-2024'
[23]:
xm.all_obs_info
[23]:
ra dec ObsID start science_usable duration proprietary_end_date revolution proprietary_usable end
0 64.925415 55.999440 0000110101 2001-08-19 07:05:23 True 0 days 09:08:33 2002-09-29 310 True 2001-08-19 16:13:56
1 263.674950 -32.581670 0001730201 2001-03-09 12:44:21 True 0 days 04:44:43 2002-05-25 229 True 2001-03-09 17:29:04
2 263.674950 -32.581670 0001730301 2001-03-09 17:30:16 True 0 days 02:36:02 2002-05-25 229 True 2001-03-09 20:06:18
3 263.674950 -32.581670 0001730401 2001-03-09 09:41:25 True 0 days 03:00:59 2002-05-25 229 True 2001-03-09 12:42:24
4 99.349995 6.135278 0001730501 2002-09-17 18:35:28 True 0 days 06:05:39 2004-12-31 508 True 2002-09-18 00:41:07
... ... ... ... ... ... ... ... ... ... ...
17556 137.211167 55.379083 0924140101 2024-04-11 19:58:54 True 0 days 04:41:40 NaT 4458 False 2024-04-12 00:40:34
17557 116.748000 -45.490722 0922040801 2024-04-14 06:14:32 True 0 days 06:43:20 NaT 4459 False 2024-04-14 12:57:52
17558 179.719375 58.583583 0924141701 2024-04-14 02:14:33 True 0 days 02:13:20 NaT 4459 False 2024-04-14 04:27:53
17559 153.948542 54.520361 0924141801 2024-04-13 19:51:03 True 0 days 05:48:20 NaT 4459 False 2024-04-14 01:39:23
17560 287.956500 4.982722 0934990101 2024-04-14 14:56:52 True 0 days 18:35:00 NaT 4459 False 2024-04-15 09:31:52

17561 rows × 10 columns

[24]:
ch.all_obs_info
[24]:
ra dec ObsID science_usable proprietary_usable start end duration proprietary_end_date target_category instrument grating data_mode
0 274.43140 -33.01883 6616 True True 2006-02-24 04:33:41.000003 2007-09-21 00:26:11.000003 573 days 19:52:30 2007-02-28 GS ACIS-S HETG CC_000A8
1 83.63292 22.01447 7587 True True 2007-02-03 09:58:57.000000 2008-07-21 21:53:57.000000 534 days 11:55:00 2008-02-06 SNR ACIS-S HETG TE_0077C
2 202.50000 47.20000 13814 True True 2012-09-20 07:21:41.999999 2012-09-22 12:47:41.999999 2 days 05:26:00 2013-10-11 NGS ACIS-S NONE TE_00958
3 266.41667 -29.00781 13842 True True 2012-07-21 11:52:41.000002 2012-07-23 17:08:41.000002 2 days 05:16:00 2012-07-25 NGS ACIS-S HETG TE_008D0
4 316.72458 38.74942 13651 True True 2012-02-13 20:18:26.999997 2012-02-16 00:51:26.999997 2 days 04:33:00 2013-02-21 MISC HRC-S LETG DEFAULT
... ... ... ... ... ... ... ... ... ... ... ... ... ...
23136 332.17000 45.74231 24644 True True 2020-09-29 01:23:47.000003 2020-09-29 01:23:47.000003 0 days 00:00:00 2020-09-30 CAL HRC-I NONE DEFAULT
23137 84.91458 -69.74361 1203 True True 1999-08-31 03:28:53.000003 1999-08-31 03:28:53.000003 0 days 00:00:00 2003-06-16 GS HRC-I NONE
23138 332.17010 45.74230 1336 True True 1999-10-03 21:48:53.000004 1999-10-03 21:48:53.000004 0 days 00:00:00 1999-12-14 CAL HRC-I NONE
23139 350.85750 58.81483 1409 True True 1999-10-23 18:29:33.999999 1999-10-23 18:29:33.999999 0 days 00:00:00 1999-12-09 SNR HRC-I NONE
23140 332.17000 45.74231 24645 True True 2020-10-25 12:05:32.000001 2020-10-25 12:05:32.000001 0 days 00:00:00 2020-10-27 CAL HRC-S NONE SCENTER

23141 rows × 13 columns

[25]:
ea.all_obs_info
[25]:
ra dec ObsID science_usable start end duration ra_min ra_max dec_min dec_max neigh_obs
0 116.703297 42.008289 117048 True 2020-04-15 10:13:39 2020-04-19 02:13:48 3 days 16:00:09 114.725275 118.681319 40.5 43.5 114045,118045,122045,113048,121048,112051,1160...
1 120.659341 42.008289 121048 True 2020-04-17 10:14:32 2020-04-21 10:13:37 3 days 23:59:05 118.681319 122.637363 40.5 43.5 118045,122045,117048,125048,116051,119051,1230...
2 124.615385 42.008289 125048 True 2020-04-19 14:14:56 2020-04-23 14:14:20 3 days 23:59:24 122.637363 126.593407 40.5 43.5 122045,126045,121048,129048,123051,127051,0000...
3 128.571429 42.008289 129048 True 2020-04-21 18:15:19 2020-04-25 18:14:56 3 days 23:59:37 126.593407 130.549451 40.5 43.5 126045,130045,125048,133048,127051,131051,0000...
4 132.527473 42.008289 133048 True 2020-04-23 22:15:47 2020-04-28 02:15:10 4 days 03:59:23 130.549451 134.505495 40.5 43.5 130045,134045,129048,136048,131051,135051,0000...
... ... ... ... ... ... ... ... ... ... ... ... ...
2442 220.000000 -87.045181 220177 True 2020-03-19 20:42:36 2020-04-03 04:43:25 14 days 08:00:49 200.000000 240.000000 -88.5 -85.5 001180,191174,214174,236174,259174,180177,2601...
2443 260.000000 -87.045181 260177 True NaT 2020-04-05 12:43:45 NaT 240.000000 280.000000 -88.5 -85.5 001180,236174,259174,281174,304174,220177,3001...
2444 300.000000 -87.045181 300177 True 2020-03-28 16:42:43 2020-04-09 04:42:26 11 days 11:59:43 280.000000 320.000000 -88.5 -85.5 001180,259174,281174,304174,326174,260177,3401...
2445 340.000000 -87.045181 340177 True 2020-03-31 20:42:37 2020-04-12 00:44:26 11 days 04:01:49 320.000000 360.000000 -88.5 -85.5 001180,304174,326174,349174,011174,300177,0201...
2446 0.000000 -90.000000 001180 True 2020-03-26 00:45:06 2020-04-06 12:45:53 11 days 12:00:47 0.000000 360.000000 -90.0 -88.5 020177,060177,100177,140177,180177,220177,2601...

2447 rows × 12 columns

[26]:
ecpv.all_obs_info
[26]:
ra dec ObsID science_usable start end duration Field_Name Field_Type active_insts
0 129.550000 1.500000 300007 True 2019-11-03 02:42:50 2019-11-04 03:36:37 89627.0 EFEDS SURVEY TM1,TM2,TM3,TM4,TM5,TM6,TM7
1 133.860000 1.500000 300008 True 2019-11-04 03:49:16 2019-11-05 05:16:39 91643.0 EFEDS SURVEY TM1,TM2,TM3,TM4,TM5,TM6,TM7
2 138.140000 1.500000 300009 True 2019-11-05 05:29:18 2019-11-06 06:40:06 90648.0 EFEDS SURVEY TM1,TM2,TM3,TM4,TM5,TM6,TM7
3 142.450000 1.500000 300010 True 2019-11-06 07:24:46 2019-11-07 08:20:08 89722.0 EFEDS SURVEY TM1,TM2,TM3,TM4,TM5,TM6,TM7
4 130.331300 -78.963400 300004 True 2019-11-16 23:14:40 2019-11-18 18:17:12 154952.0 ETA_CHA SURVEY TM1,TM2,TM3,TM4,TM5,TM6,TM7
... ... ... ... ... ... ... ... ... ... ...
165 284.146250 -37.909167 700008 True 2019-10-24 11:11:19 2019-10-25 08:55:52 78273.0 1RXS_J185635_375433 EXTRAGALACTIC_FIELDS TM1,TM2,TM3,TM4,TM5,TM6,TM7
166 281.540771 79.873726 900060 True 2019-09-24 15:27:06 2019-09-24 21:30:32 21806.0 3C390_3 EXTRAGALACTIC_FIELDS TM6
167 281.500275 79.885376 900068 True 2019-09-28 15:49:51 2019-09-28 21:30:32 20441.0 3C390_3 EXTRAGALACTIC_FIELDS TM6
168 281.489410 79.888214 900069 True 2019-09-29 15:23:24 2019-09-29 21:30:37 22033.0 3C390_3 EXTRAGALACTIC_FIELDS TM6
169 281.478271 79.891029 900070 True 2019-09-30 15:23:24 2019-09-30 21:30:32 22028.0 3C390_3 EXTRAGALACTIC_FIELDS TM5,TM6,TM7

170 rows × 10 columns

[27]:
nu.all_obs_info
[27]:
ra dec ObsID science_usable proprietary_usable start end duration proprietary_end_date target_category exposure_a exposure_b ontime_a ontime_b nupsdout issue_flag
0 83.8281 -69.2465 40001014013 True True 2013-06-29 01:16:07.183998 2013-07-05 08:51:07.184004 6 days 07:35:00.000006 2015-09-17 00:00:00 SNR 5 days 11:20:42 5 days 11:09:58 5 days 20:32:48 5 days 20:33:46 2119 1
1 83.7759 -69.2677 40001014016 True True 2014-04-22 21:06:07.183999 2014-04-29 09:31:07.183998 6 days 12:24:59.999999 2015-09-17 00:00:00 SNR 4 days 23:56:47 4 days 23:35:17 5 days 08:14:46 5 days 08:14:09 0 1
2 83.8965 -69.2477 40001014023 True True 2014-08-01 23:46:07.183998 2014-08-08 02:41:07.184000 6 days 02:55:00.000002 2015-09-17 00:00:00 SNR 4 days 22:46:56 4 days 22:35:32 5 days 07:01:07 5 days 07:02:55 0 0
3 340.6530 29.6985 60401031004 True True 2018-11-28 22:21:09.184000 2018-12-08 17:16:09.183997 9 days 18:54:59.999997 2019-12-25 00:00:00 AGN 4 days 17:35:58 4 days 17:01:43 5 days 02:48:25 5 days 02:49:43 0 0
4 344.4048 -36.9765 60901012002 True False 2023-11-02 06:06:09.184003 2023-11-10 11:41:09.184004 8 days 05:35:00.000001 2024-05-20 00:00:00 AGN 4 days 11:03:16 4 days 10:06:55 4 days 19:02:15 4 days 19:02:36 0 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
5477 0.0000 0.0000 20627025001 True True 2022-06-04 00:08:52.183997 2022-06-04 00:26:09.184004 0 days 00:17:17.000007 2022-06-13 00:00:00 SOL 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 1
5478 0.0000 0.0000 20624008001 True True 2022-06-04 00:26:09.184004 2022-06-04 02:01:09.184002 0 days 01:34:59.999998 2022-06-13 00:00:00 SOL 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 1
5479 0.0000 0.0000 20624001002 True True 2022-06-04 02:01:09.184002 2022-06-04 03:06:09.184003 0 days 01:05:00.000001 2022-06-13 00:00:00 SOL 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 1
5480 0.0000 0.0000 20601030002 True True 2022-06-04 03:06:09.184003 2022-06-04 03:26:09.184004 0 days 00:20:00.000001 2022-06-13 00:00:00 SOL 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 0
5481 0.0000 0.0000 80802021002 True True 2022-06-16 21:56:09.184001 2022-06-16 22:16:09.184002 0 days 00:20:00.000001 2023-01-11 00:00:00 TOO 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 days 00:00:00 0 0

5482 rows × 16 columns

[28]:
sw.all_obs_info
[28]:
ra dec ObsID science_usable start end duration target_category xrt_exposure bat_exposure uvot_exposure
0 328.70739 16.89364 00173780007 True 2005-12-25 00:42:01.999999 2005-12-28 00:00:23.000000 2 days 23:18:21.000001 MISC 1 days 02:44:09.014000 1 days 00:04:10 1 days 02:40:01.030000
1 39.95079 -25.21897 00131560002 True 2005-06-03 23:59:00.999998 2005-06-06 22:47:30.000002 2 days 22:48:29.000004 MISC 0 days 22:36:12.946000 0 days 22:52:03 0 days 17:46:44.297000
2 228.46115 30.87659 00164268014 True 2005-12-03 01:00:00.999996 2005-12-06 00:07:43.000000 2 days 23:07:42.000004 MISC 0 days 19:38:55.384000 0 days 19:59:15 0 days 00:00:00
3 350.64958 5.74990 00148833003 True 2005-08-06 00:26:11.999996 2005-08-09 00:50:56.000003 3 days 00:24:44.000007 MISC 0 days 19:04:07.125000 0 days 19:23:36 0 days 18:21:43.023000
4 75.27823 45.28513 00321174011 True 2008-08-31 02:25:00.999998 2008-09-03 00:34:06.999998 2 days 22:09:06 MISC 0 days 18:49:28.970000 0 days 21:08:01.650000 0 days 18:24:15.039000
... ... ... ... ... ... ... ... ... ... ... ...
353399 225.01926 -35.01827 00067092005 True 2005-01-23 13:07:01.000004 2005-01-23 23:15:55.999996 0 days 10:08:54.999992 MISC 0 days 00:00:00 0 days 02:57:51 0 days 00:00:00
353400 77.63318 -14.96722 00074164018 True 2011-05-03 12:25:59.000002 2011-05-03 17:31:53.999999 0 days 05:05:54.999997 MISC 0 days 00:00:00 0 days 00:05:31 0 days 00:00:00
353401 7.30920 -73.71531 00048706113 True 2019-03-04 23:56:36.000001 2019-03-05 00:51:04.999997 0 days 00:54:28.999996 MISC 0 days 00:00:00 0 days 00:01:01 0 days 00:00:53.499000
353402 51.88545 26.38554 03111918013 True 2022-12-25 10:22:35.999999 2022-12-25 10:32:25.999999 0 days 00:09:50 MISC 0 days 00:00:00 0 days 00:00:02 0 days 00:00:00
353403 20.08971 -5.01057 00076384008 True 2023-05-24 11:05:38.000003 2023-05-24 12:42:01.999999 0 days 01:36:23.999996 MISC 0 days 00:00:00 0 days 00:07:14 0 days 00:00:00

353404 rows × 11 columns

[29]:
ra.all_obs_info
[29]:
ra dec ObsID science_usable start end duration target_category
0 263.57088 67.500 RS930521N00 True 1990-07-11 1991-08-13 0 days 11:19:06 ASK
1 276.42533 67.500 RS930522N00 True 1990-07-11 1991-08-13 0 days 11:18:11 ASK
2 96.42533 -67.500 RS932908N00 True 1990-07-11 1991-08-13 0 days 05:47:48 ASK
3 83.57088 -67.500 RS932907N00 True 1990-07-11 1991-08-13 0 days 05:47:39 ASK
4 267.27100 61.875 RS930625N00 True 1990-07-11 1991-08-13 0 days 03:44:35 ASK
... ... ... ... ... ... ... ... ...
1373 289.08783 -61.875 RS932827N00 True 1990-09-10 1990-10-08 0 days 00:02:44 ASK
1374 350.17942 -33.750 RS932354N00 True 1990-11-09 1990-12-02 0 days 00:02:42 ASK
1375 278.17942 -61.875 RS932826N00 True 1990-09-04 1990-09-30 0 days 00:02:42 ASK
1376 273.75000 -45.000 RS932537N00 True 1990-09-07 1990-09-23 0 days 00:02:42 ASK
1377 280.46275 -50.625 RS932634N00 True 1990-09-13 1990-10-01 0 days 00:02:34 ASK

1378 rows × 8 columns

[30]:
rp.all_obs_info
[30]:
ra dec ObsID science_usable start end duration instrument with_filter target_category target_name proc_rev fits_type
0 163.1800 57.4800 RH701867A01 True 1995-04-15 23:24:16.000001 1995-05-11 14:24:47.000001 2 days 13:29:32 HRI N AGN LOCKMAN HOLE 2 RFITS V4.
1 203.6500 37.9100 RH900717N00 True 1997-06-04 16:13:00.999998 1997-07-13 22:26:45.000004 2 days 07:58:33 HRI N MISC DEEP SURVEY 2 RDF 4_0
2 163.1800 57.4800 RH701867A04 True 1997-04-15 21:51:16.999998 1997-04-28 16:37:45.999998 2 days 06:33:41 HRI N AGN LOCKMAN HOLE 2 RFITS V3.
3 350.8700 58.8100 RH500444N00 True 1995-12-23 22:18:36.999999 1996-02-01 10:17:53.999998 2 days 02:07:27 HRI N SNR CAS A 2 RDF 3_4
4 163.1800 57.4800 RH701867A02 True 1996-05-01 02:09:33.000002 1996-05-29 15:31:24.999997 2 days 01:05:24 HRI N AGN LOCKMAN HOLE 2 RFITS V4.
... ... ... ... ... ... ... ... ... ... ... ... ... ...
11426 84.2900 -80.4700 RP999998A02 False 1991-10-20 04:07:51.999998 1991-11-01 22:43:43.000003 0 days 00:00:00 PSPCB N EGE Idle Point 2 RDF 3_4
11427 93.1800 -81.8300 RH150094N00 False 1990-07-29 22:13:42.999997 1990-07-29 22:56:58.999998 0 days 00:00:00 HRI N STR Calibration Source 2 RFITS V4.
11428 218.1540 -44.2031 RH001034N00 False 1997-08-28 00:29:42.000000 1997-08-28 01:09:58.999997 0 days 00:00:00 HRI N MISC 2 RDF 4_2
11429 258.1383 -23.3850 RH800067M01 False 1991-03-19 20:33:48.999997 1991-03-19 20:35:59.999997 0 days 00:00:00 HRI N GCL OPHIUCHUS CLUSTER 2 RFITS V3.
11430 205.4270 -62.3400 RH001027N00 False 1997-08-25 04:54:58.000003 1997-08-25 05:18:28.000000 0 days 00:00:00 HRI N MISC 2 RDF 4_2

11431 rows × 13 columns

[31]:
su.all_obs_info
[31]:
ra dec ObsID science_usable start end duration target_category xis0_expo xis0_num_modes xis1_expo xis1_num_modes xis2_expo xis2_num_modes xis3_expo xis3_num_modes
0 91.1523 -86.6779 701018010 True 2006-04-13 16:24:07.999998 2006-04-14 01:52:19.000001 0 days 09:28:11.000003 EGS 0 days 05:29:41.700000 2 0 days 05:29:41.700000 2 0 days 05:29:25.700000 2 0 days 05:29:33.700000 2
1 48.9864 -85.5003 404019010 True 2009-07-16 14:26:31.000001 2009-07-18 03:30:15.999998 1 days 13:03:44.999997 GS 0 days 17:31:24.300000 2 0 days 17:31:24.300000 2 0 days 00:00:00 0 0 days 17:31:24.300000 2
2 239.2836 -79.2302 703059010 True 2008-10-13 15:35:23.999997 2008-10-15 21:37:21.000000 2 days 06:01:57.000003 EGS 0 days 22:49:57 2 0 days 22:49:43.500000 2 0 days 00:00:00 0 0 days 22:49:35.500000 2
3 265.9438 -76.3446 705013010 True 2010-04-14 00:16:05.000002 2010-04-14 16:45:11.000002 0 days 16:29:06 EGS 0 days 11:46:23.600000 2 0 days 11:46:18.300000 2 0 days 00:00:00 0 0 days 11:46:29 2
4 74.6207 -75.2810 404036010 True 2009-06-14 01:51:18.000003 2009-06-16 13:27:19.000002 2 days 11:36:00.999999 GS 1 days 05:42:37.800000 2 1 days 05:42:37.800000 2 0 days 00:00:00 0 1 days 05:42:29.800000 2
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
3036 247.6537 82.9256 706004010 True 2011-04-16 00:38:47.000003 2011-04-16 10:23:15.000000 0 days 09:44:27.999997 EGS 0 days 05:13:55.700000 3 0 days 05:13:42 2 0 days 00:00:00 0 0 days 05:13:55.600000 2
3037 116.7261 85.6829 804031010 True 2010-02-08 09:53:43.999999 2010-02-08 18:15:19.000002 0 days 08:21:35.000003 GCL 0 days 05:19:56 1 0 days 05:19:56 1 0 days 00:00:00 0 0 days 05:19:56 1
3038 112.8158 85.6965 804030010 True 2010-02-08 18:18:12.000001 2010-02-09 12:00:25.000004 0 days 17:42:13.000003 GCL 0 days 10:17:39.100000 2 0 days 10:17:39.100000 2 0 days 00:00:00 0 0 days 10:17:39.100000 2
3039 112.8340 85.6974 804030020 True 2010-02-10 04:18:27.999997 2010-02-11 06:00:24.000002 1 days 01:41:56.000005 GCL 0 days 14:36:01.100000 2 0 days 14:36:01.100000 2 0 days 00:00:00 0 0 days 14:36:01.100000 2
3040 264.6080 87.3047 705012010 True 2010-04-26 23:41:56.000000 2010-04-27 10:16:15.000001 0 days 10:34:19.000001 EGS 0 days 05:37:59.600000 2 0 days 05:37:59.600000 2 0 days 00:00:00 0 0 days 05:37:59.600000 2

3041 rows × 16 columns

[32]:
asca.all_obs_info
[32]:
ra dec ObsID science_usable start end duration target_category sis_exposure gis_exposure
0 288.0116 5.0542 48002000 True 2000-03-23 05:52:13.440 2000-04-04 00:00:25.920 11 days 18:08:12.480000 XRB 5 days 07:25:04 5 days 14:47:28
1 253.5540 39.8158 78002000 True 2000-03-01 12:08:12.480 2000-03-11 00:00:51.840 9 days 11:52:39.360000 AGN 4 days 00:45:04 5 days 04:32:16
2 14.3115 -22.4431 77036000 True 1999-12-03 11:50:38.400 1999-12-15 19:00:28.800 12 days 07:09:50.400000 AGN 4 days 08:14:08 5 days 00:42:40
3 203.8856 -34.3378 77003000 True 1999-07-19 00:40:19.200 1999-07-29 15:40:27.840 10 days 15:00:08.640000 AGN 4 days 02:03:12 4 days 16:33:20
4 266.4864 -28.9429 48004000 True 2000-03-11 00:08:29.760 2000-03-22 00:00:43.200 10 days 23:52:13.440000 XRB 3 days 22:37:04 4 days 07:30:24
... ... ... ... ... ... ... ... ... ... ...
3069 87.7568 -32.2321 77010010 True 1999-10-11 13:47:08.160 1999-10-11 14:19:32.160 0 days 00:32:24 AGN 0 days 00:06:40 0 days 00:06:40
3070 150.6550 -58.6295 15000090 True 1993-08-05 18:31:58.080 1993-08-05 19:12:43.200 0 days 00:40:45.120000 GS 0 days 00:06:56 0 days 00:05:52
3071 335.1555 -24.7349 66013000 True 1998-11-16 16:24:48.960 1998-11-18 17:34:39.360 2 days 01:09:50.400000 NGS 0 days 14:38:24 0 days 00:00:00
3072 333.8738 -17.8223 77002000 True 1999-11-17 04:45:41.760 1999-11-19 01:35:54.240 1 days 20:50:12.480000 AGN 0 days 16:30:24 0 days 00:00:00
3073 244.9340 -15.8493 40022010 True 1993-08-16 08:40:42.240 1993-08-16 10:20:21.120 0 days 01:39:38.880000 XRB 0 days 00:53:04 0 days 00:00:00

3074 rows × 10 columns

[33]:
inte.all_obs_info
[33]:
ra dec ObsID science_usable proprietary_usable start end duration target_category jemx1_exposure jemx2_exposure isgri_exposure picsit_exposure spi_exposure scw_ver
0 294.97232 -89.65222 218500510010 True True 2020-01-22 20:40:39.751343 2020-01-22 22:17:46.754676 0 days 01:37:07.003333 MISC 0 days 01:36:12 0 days 01:36:12 0 days 01:36:10 0 days 00:28:29 0 days 01:36:12 001
1 117.47166 -89.57144 218600510010 True True 2020-01-25 11:40:02.881811 2020-01-25 12:38:50.883829 0 days 00:58:48.002018 MISC 0 days 00:57:57 0 days 00:57:57 0 days 00:57:55 0 days 00:57:27 0 days 00:57:57 001
2 166.19875 -89.54264 229200810010 True True 2020-11-02 08:42:24.928042 2020-11-02 09:41:14.930022 0 days 00:58:50.001980 MISC 0 days 00:58:00 0 days 00:58:00 0 days 00:57:44 0 days 00:57:29 0 days 00:58:00 001
3 130.24533 -89.51414 209300110010 True True 2019-05-21 07:51:13.039554 2019-05-21 08:24:33.041123 0 days 00:33:20.001569 MISC 0 days 00:32:25 0 days 00:32:25 0 days 00:32:23 0 days 00:32:04 0 days 00:32:25 001
4 125.61337 -89.47669 209300370010 True True 2019-05-21 22:51:03.082145 2019-05-21 23:24:24.083715 0 days 00:33:21.001570 MISC 0 days 00:32:30 0 days 00:32:30 0 days 00:32:28 0 days 00:32:02 0 days 00:32:30 001
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
198281 341.90903 88.90408 260500570010 True True 2023-02-12 08:47:47.388930 2023-02-12 09:17:47.390084 0 days 00:30:00.001154 MISC 0 days 00:29:11 0 days 00:29:11 0 days 00:29:10 0 days 00:28:44 0 days 00:29:11 001
198282 153.50116 88.91008 260500660010 True True 2023-02-12 13:36:15.400319 2023-02-12 14:06:15.401552 0 days 00:30:00.001233 MISC 0 days 00:28:40 0 days 00:28:40 0 days 00:29:09 0 days 00:00:00 0 days 00:29:10 001
198283 213.64879 89.06850 260900540010 True True 2023-02-22 18:05:50.978884 2023-02-22 19:29:20.982045 0 days 01:23:30.003161 MISC 0 days 01:21:28 0 days 01:21:00 0 days 01:21:09 0 days 00:28:33 0 days 01:20:57 001
198284 268.11816 89.32161 260700500010 True True 2023-02-17 10:30:57.695328 2023-02-17 11:00:58.696476 0 days 00:30:01.001148 MISC 0 days 00:29:10 0 days 00:29:10 0 days 00:29:10 0 days 00:28:47 0 days 00:29:12 001
198285 6.62067 89.56795 261300370010 True True 2023-03-05 00:34:23.640085 2023-03-05 01:04:23.641170 0 days 00:30:00.001085 MISC 0 days 00:29:04 0 days 00:29:04 0 days 00:29:48 0 days 00:29:19 0 days 00:29:52 001

198286 rows × 15 columns

Filter Array

This is unlikely to ever be accessed directly by the user, but is what defines the observations that a mission currently deems to be accepted/selected. It is a boolean numpy array with a length equal to the number of observations in the all_obs_info dataframe, a True value means the observation is accepted and a False value means it is excluded; all observations start off as accepted.

Various filtering methods can be used to adjust the filter array and set the observations which are to be downloaded/included in a DAXA archive, depending on your particular sample and science case.

It is also possible to manually set this filter array, as is demonstrated below:

[34]:
# The filter array defaults to all True, so all observations are accepted
xm.filter_array
[34]:
array([ True,  True,  True, ...,  True,  True,  True])
[35]:
# Demonstrating manually setting a filter array - it must be boolean and be the same length as the
#  'all_obs_info' table, otherewise it will not be accepted
demo_filt_arr = np.full(len(xm.all_obs_info), True)
demo_filt_arr[0] = False
xm.filter_array = demo_filt_arr
xm.filter_array
[35]:
array([False,  True,  True, ...,  True,  True,  True])

Selecting relevant observations

Few users will wish to download, process, and maintain complete observation archives, preferring to just locate data that may be relevant to the sources that they are studying. This can be achieved with the use of several filtering methods which are built into all DAXA missions.

Here we introduce the different filtering methods that are currently implemented for DAXA missions, but we do not provide detailed demonstrations of their use; that is left to specific case studies designed to show scientists with different needs how DAXA can be used in ways that are most relevant to them.

Filtering on ObsID

The most basic filtering method available can be used when you already know which observation(s) you are interested in - if you have the ObsID(s) you can just pass them to the filter_on_obs_ids method, and select only that data:

[36]:
help(xm.filter_on_obs_ids)
Help on method filter_on_obs_ids in module daxa.mission.base:

filter_on_obs_ids(allowed_obs_ids: Union[str, List[str]]) method of daxa.mission.xmm.XMMPointed instance
    This filtering method will select only observations with IDs specified by the allowed_obs_ids argument.

    Please be aware that filtering methods are cumulative, so running another method will not remove the
    filtering that has already been applied, you can use the reset_filter method for that.

    :param str/List[str] allowed_obs_ids: The ObsID (or list of ObsIDs) that you wish to be let
        through the filter.

Filtering on position

Arguably the most useful type of filtering supported by DAXA missions, the filter_on_positions method allows us to search for observations that are relevant to specific positions on the sky (generally these will represent particular objects). A search can be performed either for a single position, or for a whole sample.

The search_distance argument controls how close the central coordinate of an observation must be to a search position for that observation to be accepted. The default search positions are defined by the field-of-view of the telescope (if a mission has multiple instruments with different field-of-views then each instrument will be searched with the correct field-of-view). The user may also specify their own search distance value (or values), and it is also possible to specify a different search distance for every position.

The return_pos_obs_info argument controls whether a dataframe is returned that links passed positions to particular observations that are relevant to them - this dataframe would not include positions that are determined to have no relevant observations.

[37]:
help(xm.filter_on_positions)
Help on method filter_on_positions in module daxa.mission.base:

filter_on_positions(positions: Union[list, numpy.ndarray, astropy.coordinates.sky_coordinate.SkyCoord], search_distance: Union[astropy.units.quantity.Quantity, float, int, list, numpy.ndarray, dict] = None, return_pos_obs_info: bool = False) -> Optional[pandas.core.frame.DataFrame] method of daxa.mission.xmm.XMMPointed instance
    This method allows you to filter the observations available for a mission based on a set of coordinates for
    which you wish to locate observations. The method searches for observations by the current mission that have
    central coordinates within the distance set by the search_distance argument.

    Please be aware that filtering methods are cumulative, so running another method will not remove the
    filtering that has already been applied, you can use the reset_filter method for that.

    :param list/np.ndarray/SkyCoord positions: The positions for which you wish to search for observations. They
        can be passed either as a list or nested list (i.e. [r, d] OR [[r1, d1], [r2, d2]]), a numpy array, or
        an already defined SkyCoord. If a list or array is passed then the coordinates are assumed to be in
        degrees, and the default mission frame will be used.
    :param Quantity/float/int/list/np.ndarray/dict search_distance: The distance within which to search for
        observations by this mission. Distance may be specified either as an Astropy Quantity that can be
        converted to degrees (a float/integer will be assumed to be in units of degrees), as a dictionary of
        quantities/floats/ints where the keys are names of different instruments (possibly with different field
        of views), or as a non-scalar Quantity, list, or numpy array with one entry per set of coordinates (for
        when you wish to use different search distances for each object). The default is None, in which case a
        value of 1.2 times the approximate field of view defined for each instrument will be used; where different
        instruments have different FoVs, observation searches will be undertaken on an instrument-by-instrument
        basis using the different field of views.
    :param bool return_pos_obs_info: Allows this method to return information (in the form of a Pandas dataframe)
        which identifies the positions which have been associated with observations, and the observations they have
        been associated with. Default is False.
    :return: If return_pos_obs_info is True, then a dataframe containing information on which ObsIDs are relevant
        to which positions will be returned. If return_pos_obs_info is False, then None will be returned.
    :rtype: Union[None,pd.DataFrame]

Filtering on name

If you are interested in data relevant to named objects, you can pass the name(s) to the filter_on_name method. It will use a lookup service (specifically Sesame), to locate coordinates for the object(s), and then pass that to the filter_on_positions method.

Bear in mind that you are reliant on the lookup service having accurate central coordinates for the named object, so it could be worth checking with your own coordinates and using filter_on_positions directly if you can’t find any observations!

[38]:
help(xm.filter_on_name)
Help on method filter_on_name in module daxa.mission.base:

filter_on_name(object_name: Union[str, List[str]], search_distance: Union[astropy.units.quantity.Quantity, float, int, list, numpy.ndarray, dict] = None, parse_name: bool = False) method of daxa.mission.xmm.XMMPointed instance
    This method wraps the 'filter_on_positions' method, and allows you to filter the mission's observations so
    that it contains data on a single (or a list of) specific objects. The names are passed by the user, and
    then parsed into coordinates using the Sesame resolver. Those coordinates and the search distance are
    then used to find observations that might be relevant.

    :param str/List[str] object_name: The name(s) of objects you would like to search for.
    :param Quantity/float/int/list/np.ndarray/dict search_distance: The distance within which to search for
        observations by this mission. Distance may be specified either as an Astropy Quantity that can be
        converted to degrees (a float/integer will be assumed to be in units of degrees), as a dictionary of
        quantities/floats/ints where the keys are names of different instruments (possibly with different field
        of views), or as a non-scalar Quantity, list, or numpy array with one entry per set of coordinates (for
        when you wish to use different search distances for each object). The default is None, in which case a
        value of 1.2 times the approximate field of view defined for each instrument will be used; where different
        instruments have different FoVs, observation searches will be undertaken on an instrument-by-instrument
        basis using the different field of views.
    :param bool parse_name: Whether to attempt extracting the coordinates from the name by parsing with a regex.
        For objects catalog names that have J-coordinates embedded in their names, e.g.,
        'CRTS SSS100805 J194428-420209', this may be much faster than a Sesame query for the same object name.

Filtering on target type

It is possible to filter observations based on the type of object that was the original target of the observation. Warning: this should be used with significant caution, as our object taxonomy may not be granular enough to represent all different types of astronomical objects, and conversions between the different target types used by different missions and our target types is imperfect!

Here we display the DAXA source type taxonomy, the short form codes on the left are what should be passed to the filter_on_target_type method - the user may pass either a single target type, or a list of them:

[39]:
xm.show_allowed_target_types()
╒═══════════════╤═══════════════════════════════════════════════╕
│ Target Type   │ Description                                   │
╞═══════════════╪═══════════════════════════════════════════════╡
│ AGN           │ Active Galaxies and Quasars                   │
├───────────────┼───────────────────────────────────────────────┤
│ BLZ           │ Blazars                                       │
├───────────────┼───────────────────────────────────────────────┤
│ CV            │ Cataclysmic Variables                         │
├───────────────┼───────────────────────────────────────────────┤
│ CAL           │ Calibration Observation (possibly of objects) │
├───────────────┼───────────────────────────────────────────────┤
│ EGS           │ Extragalactic Surveys                         │
├───────────────┼───────────────────────────────────────────────┤
│ GCL           │ Galaxy Clusters                               │
├───────────────┼───────────────────────────────────────────────┤
│ GS            │ Galactic Survey                               │
├───────────────┼───────────────────────────────────────────────┤
│ ASK           │ All Sky Survey                                │
├───────────────┼───────────────────────────────────────────────┤
│ MAG           │ Magnetars and Rotation-Powered Pulsars        │
├───────────────┼───────────────────────────────────────────────┤
│ NGS           │ Normal and Starburst Galaxies                 │
├───────────────┼───────────────────────────────────────────────┤
│ NS            │ Neutron stars and Black Holes                 │
├───────────────┼───────────────────────────────────────────────┤
│ STR           │ Non-degenerate and White Dwarf Stars          │
├───────────────┼───────────────────────────────────────────────┤
│ OAGN          │ Obscured Active Galaxies and Quasars          │
├───────────────┼───────────────────────────────────────────────┤
│ SNE           │ Non-ToO Supernovae                            │
├───────────────┼───────────────────────────────────────────────┤
│ SNR           │ Supernova Remnants and Galactic diffuse       │
├───────────────┼───────────────────────────────────────────────┤
│ SOL           │ Solar System Observations                     │
├───────────────┼───────────────────────────────────────────────┤
│ ULX           │ Ultra-luminous X-ray Sources                  │
├───────────────┼───────────────────────────────────────────────┤
│ XRB           │ X-ray Binaries                                │
├───────────────┼───────────────────────────────────────────────┤
│ TOO           │ Targets of Opportunity                        │
├───────────────┼───────────────────────────────────────────────┤
│ EGE           │ Extended galactic or extragalactic            │
├───────────────┼───────────────────────────────────────────────┤
│ MISC          │ Catch-all for other sources                   │
╘═══════════════╧═══════════════════════════════════════════════╛
[40]:
help(xm.filter_on_target_type)
Help on method filter_on_target_type in module daxa.mission.base:

filter_on_target_type(target_type: Union[str, List[str]]) method of daxa.mission.xmm.XMMPointed instance
    This method allows the filtering of observations based on what type of object their target source was. It
    is only supported for missions that have that data available, and will raise an exception for those
    missions that don't support this filtering.

    WARNING: You should not trust these target types without question, they are the result of crude mappings, and
    some may be incorrect. They also don't take into account sources that might serendipitously appear in
    a particular observation.

    :param str/List[str] target_type: The types of target source you would like to find observations of. For
        allowed types, please use the 'show_allowed_target_types' method. Can either be a single type, or
        a list of types.

Filtering on time

Observations can be filtered on when they were taken, with the user specifying a time frame (defined by a start and end date-time) from which they wish to select observations. By default any observation that coincides with that time window (either starting in it, ending in it, or starting and ending outside but being taken during it) will be selected - it is also possible to require that an observation must have taken place entirely within the time window.

Warning: Observations of survey missions like eRASS1DE and ROSATAllSky may repeatedly visit a particular location, and all that data may be incorporated as one observation, meaning that there may not constant coverage in such an observation

[41]:
help(xm.filter_on_time)
Help on method filter_on_time in module daxa.mission.base:

filter_on_time(start_datetime: datetime.datetime, end_datetime: datetime.datetime, over_run: bool = True) method of daxa.mission.xmm.XMMPointed instance
    This method allows you to filter observations for this mission based on when they were taken. A start
    and end time are passed by the user, and observations that fall within that window are allowed through
    the filter. The exact behaviour of this filtering method is controlled by the over_run argument, if set
    to True then observations with a start or end within the search window will be selected, but if False
    then only observations with a start AND end within the window are selected.

    Please be aware that filtering methods are cumulative, so running another method will not remove the
    filtering that has already been applied, you can use the reset_filter method for that.

    :param datetime start_datetime: The beginning of the time window in which to search for observations.
    :param datetime end_datetime: The end of the time window in which to search for observations.
    :param bool over_run: This controls whether selected observations have to be entirely within the passed
        time window or whether either a start or end time can be within the search window. If set
        to True then observations with a start or end within the search window will be selected, but if False
        then only observations with a start AND end within the window are selected. Default is True.

Filtering on time & position

A method of filtering that combines positional and temporal filtering, so that observations of a particular position, within a particular time window, can be located. The user does not have to filter for one position-time combination at a time - they may pass a set of positions, with a corresponding set of time windows, and find all the observations that fulfill those requirements.

The filter_on_positions_at_time method supports the same arguments passed to filter_on_positions and filter_on_time, which set search distances from the passed coordinate (search_distance), whether a dataframe linking particular input coordinates to particular selected observations (return_obs_info), and whether only observations that start and end within the specified time period should be considered (over_run).

[42]:
help(xm.filter_on_positions_at_time)
Help on method filter_on_positions_at_time in module daxa.mission.base:

filter_on_positions_at_time(positions: Union[list, numpy.ndarray, astropy.coordinates.sky_coordinate.SkyCoord], start_datetimes: Union[numpy.ndarray, datetime.datetime], end_datetimes: Union[numpy.ndarray, datetime.datetime], search_distance: Union[astropy.units.quantity.Quantity, float, int, list, numpy.ndarray, dict] = None, return_obs_info: bool = False, over_run: bool = True) method of daxa.mission.xmm.XMMPointed instance
    This method allows you to filter the observations available for a mission based on a set of coordinates for
    which you wish to locate observations that were taken within a certain time frame. The method spatially
    searches for observations that have central coordinates within the distance set by the search_distance
    argument, and temporally by start and end times passed by the user; and observations that fall within that
    window are allowed through the filter.

    The exact behaviour of the temporal filtering method is controlled by the over_run argument, if set
    to True then observations with a start or end within the search window will be selected, but if False
    then only observations with a start AND end within the window are selected.

    Please be aware that filtering methods are cumulative, so running another method will not remove the
    filtering that has already been applied, you can use the reset_filter method for that.

    :param list/np.ndarray/SkyCoord positions: The positions for which you wish to search for observations. They
        can be passed either as a list or nested list (i.e. [r, d] OR [[r1, d1], [r2, d2]]), a numpy array, or
        an already defined SkyCoord. If a list or array is passed then the coordinates are assumed to be in
        degrees, and the default mission frame will be used.
    :param np.array(datetime)/datetime start_datetimes: The beginnings of time windows in which to search for
        observations. There should be one entry per position passed.
    :param np.array(datetime)/datetime end_datetimes: The endings of time windows in which to search for
        observations. There should be one entry per position passed.
    :param Quantity/float/int/list/np.ndarray/dict search_distance: The distance within which to search for
        observations by this mission. Distance may be specified either as an Astropy Quantity that can be
        converted to degrees (a float/integer will be assumed to be in units of degrees), as a dictionary of
        quantities/floats/ints where the keys are names of different instruments (possibly with different field
        of views), or as a non-scalar Quantity, list, or numpy array with one entry per set of coordinates (for
        when you wish to use different search distances for each object). The default is None, in which case a
        value of 1.2 times the approximate field of view defined for each instrument will be used; where different
        instruments have different FoVs, observation searches will be undertaken on an instrument-by-instrument
        basis using the different field of views.
    :param bool return_obs_info: Allows this method to return information (in the form of a Pandas dataframe)
        which identifies the positions which have been associated with observations, in the specified time
        frame, and the observations they have been associated with. Default is False.
    :param bool over_run: This controls whether selected observations have to be entirely within the passed
        time window or whether either a start or end time can be within the search window. If set
        to True then observations with a start or end within the search window will be selected, but if False
        then only observations with a start AND end within the window are selected. Default is True.

Downloading data

Once the user has decided upon a set of observations, for a particular mission, that are relevant to their research - the next step is often to download them. This section specifies what can be downloaded, and how to download it.

What can be downloaded?

The exact data that can be downloaded depends on what a particular mission has made available on their online archive - Event lists are available for all missions bar INTEGRAL, and most missions support the acquisition of pre-generated images, but not all (INTEGRAL and eROSITA CalPV) for instance).

We make a distinction between ‘raw’ and ‘pre-processed’ data, which is necessarily fuzzy due to the disparate natures of the various data archives we have to deal with. Broadly speaking ‘raw’ data means either absolutely raw files that need to be processed into initial event lists (the case with XMMPointed) or just event lists (uncleaned and pre-cleaned); ‘pre-processed’ data includes pre-processed data products such as images, exposure maps, and background maps.

This means that the user can either set up an archive using the pre-processed data, or re-process data using DAXA interfaces to the various telescope backend software packages.

Pre-processed products are downloaded by default (in addition to the raw data), but when setting up an Archive you can choose between using pre-processed data or re-processing the raw data; if you do not wish to acquire pre-processed data products, you can pass download_products=False to the download method (see below).

Event lists are considered raw data (i.e. not pre-processed) for most missions, and will always be downloaded

XMM-Newton

Currently we only support the acquisition of raw data.

Chandra

The following data products can be downloaded:

  • Full FoV image (if an Archive is constructed including pre-processed Chandra data, this is what is included).

  • High-resolution central image.

eROSITA All-Sky DR1 (German Half)

The following data products can be downloaded:

  • Images

  • Exposure maps

  • Background maps

eROSITA Calibration and Performance Verification

No pre-processed data products are available, only event lists.

NuSTAR-Pointed

The following data products can be downloaded:

  • Images

Swift

The following data products can be downloaded (no products are available for BAT, only raw data):

  • Images (XRT and UVOT)

  • Exposure maps (XRT and UVOT)

ROSAT All-Sky Survey

The following data products can be downloaded:

  • Images

  • Exposure maps

ROSAT-Pointed

The following data products can be downloaded:

  • Images

  • Exposure maps (only PSPC, no HRI)

Suzaku

The following data products can be downloaded:

  • Images

ASCA

The following data products can be downloaded:

  • Images (SIS0 + SIS1, and GIS2 + GIS3)

  • Exposure maps

  • Lightcurves (not included in Archive constructed from pre-processed data)

  • Spectra (not included in Archive constructed from pre-processed data)

INTEGRAL-Pointed

No pre-processed data (nor even event lists) are available to download for INTEGRAL, only raw data and calibration files.

How can it be downloaded?

The selected data can be downloaded using the download method that is built into each mission class.

If multiple observations have been selected, then downloads can be multi-threaded; this can sometimes offer a speed benefit, though only if the archive and user internet connections are not saturated by the download. The number of cores used can be set via the num_cores argument; by defualt this is set to the NUM_CORES DAXA constant, which can either be set by the user in the DAXA configuration file/by setting the value of daxa.NUM_CORES, or will be 90% of the cores of the system.

Note that it is not necessary to manually activate the download method if you will be creating a DAXA Archive from the filtered mission objects, as that will be done on archive initialisation -see the Archive tutorial for more information.

Here we demonstrate a simple XMM download, as well as a Chandra download that includes pre-generated images (standard Chandra reprocessing scripts can be used on this downloaded data):

[43]:
xm.filter_on_obs_ids('0201903501')
xm.download(num_cores=1)

ch.filter_on_obs_ids('3205')
ch.download(num_cores=1, download_products=True)
Downloading XMM-Newton Pointed data: 100%|██████████████████████████████████████| 1/1 [00:32<00:00, 32.13s/it]
Downloading Chandra data: 100%|█████████████████████████████████████████████████| 1/1 [00:31<00:00, 31.23s/it]

Getting paths to downloaded mission data products

If pre-processed data have been downloaded, it is possible to use methods built into the mission class to retrieve the paths to various data products. If you intend on using an XGA archive, using these methods should not be necessary as the data will be moved to the Archive processed data storage structure, but it may still be useful if you just wish to use DAXA to download data.

There are four different get methods:

  • get_evt_list_path - to retrieve event lists.

  • get_image_path - to retrieve images.

  • get_expmap_path - to retrieve exposure maps.

  • get_background_path - to retrieve background maps.

They are very easy to use - only the ObsID, instrument (in most cases, not necessary if the mission has only one instrument per ObsID), and lower/upper energy bound (for images, exposure maps, and background maps - though if the products only have one energy band it will be completed automatically) need to be provided. They also provide helpful, detailed, error messages if what is requested isn’t possible.

Here we retrive the path to the event list (note that we do not need to pass the instrument name, as Chandra has only one per ObsID):

[44]:
ch.get_evt_list_path('3205')
[44]:
'/Users/dt237/code/DAXA/docs/source/notebooks/tutorials/daxa_output/chandra_raw/3205/primary/acisf03205N006_evt2.fits'

We can also retrieve the path to the full-FoV image (again we don’t need to pass instrument in this specific instance, nor do we need to pass energy bounds because Chandra only supplies one):

[45]:
ch.get_image_path('3205')
[45]:
'/Users/dt237/code/DAXA/docs/source/notebooks/tutorials/daxa_output/chandra_raw/3205/primary/acisf03205N006_full_img2.fits'

A more typical way of using this function would be this:

[46]:
ch.get_image_path('3205', Quantity(0.5, 'keV'), Quantity(7, 'keV'), 'ACIS-I')
[46]:
'/Users/dt237/code/DAXA/docs/source/notebooks/tutorials/daxa_output/chandra_raw/3205/primary/acisf03205N006_full_img2.fits'

The get methods for exposure and background map paths operate identically to the image path method, though will show an error for Chandra:

[47]:
ch.get_expmap_path('3205', Quantity(0.5, 'keV'), Quantity(7, 'keV'), 'ACIS-I')
---------------------------------------------------------------------------
PreProcessedNotSupportedError             Traceback (most recent call last)
Cell In [47], line 1
----> 1 ch.get_expmap_path('3205', Quantity(0.5, 'keV'), Quantity(7, 'keV'), 'ACIS-I')

File ~/code/DAXA/daxa/mission/base.py:2034, in BaseMission.get_expmap_path(self, obs_id, lo_en, hi_en, inst)
   2022 """
   2023 A get method that provides the path to a downloaded pre-generated exposure map for the current mission (if
   2024 available). This method will not work if pre-processed data have not been downloaded.
   (...)
   2031 :rtype: str
   2032 """
   2033 if self._template_exp_name is None:
-> 2034     raise PreProcessedNotSupportedError("This mission ({m}) does not support the download of pre-processed "
   2035                                         "exposure maps, so a path cannot be "
   2036                                         "provided.".format(m=self.pretty_name))
   2038 if lo_en is not None:
   2039     # We make sure that the provided energy bounds are in keV
   2040     lo_en = lo_en.to('keV')

PreProcessedNotSupportedError: This mission (Chandra) does not support the download of pre-processed exposure maps, so a path cannot be provided.

Saving mission state

It is possible to save the current ‘state’ of a mission object to a file - which in turn can be used to reload a mission class and have it be in that same state. This is primarily used by internal DAXA methods, to allow saved archives to reinstate their constituent mission classes when they are loaded back in.

However, it is also possible to do this manually, in order to reload missions as they were when they were saved. The save file could also be shared with other users, to allow them to create a mission in the same state.

The state of the mission includes the filtering procedures performed upon it, and the current filtered data - this means that the observations selected when the mission was saved will be reloaded exactly, and the same filtering steps can be re-applied on reload to update the mission (if more observations have become available).

Saving a mission state is simple - call the save() method and pass a root storage directory and optionally a file name (if this is not supplied the save file will be the mission name + ‘_state.json’):

[48]:
xm.save("demo_save/", "demo_xmm_save.json")

A mission can then be reinstated from the file on declaration:

[49]:
reinstated_xm = XMMPointed(save_file_path="demo_save/demo_xmm_save.json")
reinstated_xm.filtered_obs_info
/Users/dt237/code/DAXA/daxa/mission/xmm.py:83: UserWarning: 140 of the 17701 observations located for this mission have been removed due to NaN RA or Dec values
  self._fetch_obs_info()
[49]:
ra dec ObsID start science_usable duration proprietary_end_date revolution proprietary_usable end
3527 149.592285 -11.0597 0201903501 2004-06-17 10:38:28 True 0 days 04:08:33 2004-06-17 828 True 2004-06-17 14:47:01

If some time has passed since the mission state was saved, and you wish to re-run the original filtering procedures to check if new data are available, you can call the update_filtering() method.