Documentation
SANEPIC Documentation
This page present quickly the SANEPIC Workflow and well as its input data and files format.
Workflow
SANEPIC is organized in several programs to be used in a comprehensive framework :
- sanePre : is used to perform preprocessing steps.
- sanePos : will take care of all computations with regards to astronomical positions
- saneInv : will inverse the noise-noise power spectra (see Patanchon et al. 2008)
- sanePic : is the main program where the complete inversion of the map-making equation is performed.
All the different programs use a central configuration file and some other input files. The workflow of SANEPIC is summarized with this graphic :
The input data format is described below, a well as the output maps files.
Data Format
For the time ordered data, SANEPIC use a simple, multi-extension, fits file. Each fits file, or frame, describe a continuous set of data as :
- signal & mask are two mandatory 2D images, with dimensions nTime x nBolo, coded in 64 and 8 bits respectively.
-
the positions of the bolometers on the sky, is also mandatory and can be provided in two different ways :
- as lon & lat, two 2D images with dimensions nTime x nBolo,
- or as a reference position and bolometers offsets, two binary tables with 3 columns each,
- channels, a binary table containing at least one columns, names, and nBolo lines naming each bolometers.
- time, a 1D image, with nTime items, can also be provided, in order to perform some checks.
Thus, a typical nTime=152612 and nBolo=41 dataset (108 MB) would look like :
Primary header |
132 records No data |
|
---|---|---|
Extension 1 | signal |
Header : 25 records IMAGE ( 152612 x 41 ) |
Extension 2 | lon |
Header : 27 records IMAGE ( 152612 x 41 ) |
Extension 3 | lat |
Header : 27 records IMAGE ( 152612 x 41 ) |
Extension 4 | mask |
Header : 19 records IMAGE ( 152612 x 41 ) |
Extension 5 | time |
Header : 24 records IMAGE ( 152612 ) |
Extension 6 | channels |
Header : 52 records Binary Table ( 6 x 41 ) |
Extension 7 | reference position |
Header : 52 records Binary Table ( 24 x 152612 ) |
Extension 8 | offsets |
Header : 43 records Binary Table ( 22 x 41 ) |
Note that the extensions are accessed by their names (EXTNAM keyword), and not by their position/number in the fits file.
Ini Files
The input file of SANEPIC follow the ini format. It consists of several sections describing the input of each programs : (complete example file)
The common section is used to describe the fields used by all programs, mainly input output and temporary directories, lists of fits files and bolometers
[common] | ||||
data_directory | = | ./data/ | ; | source data directory |
input_directory | = | ./inputDir/ | ; | input directory with all the configurations files |
output_dir | = | ./Output/ | ; | output directory |
temp_dir | = | ./tempDir/ | ; | temporary directory |
fits_filelist | = | listFile.list | ; | file containing fits file names |
bolo_global_file | = | ; | every scans have the same detector list | |
bolo_suffix | = | .bolo | ; | bolometers filelist suffix |
The format of the input files and an explanation of the suffix mechanism will follow
All quantities related to positions and map will be here, pixel size, projection center, projection type, coordinate type, coordinates transformation (galactic/equatorial), file format, masks and mapping of flagged data
[sanePos] | ||||
pixsize | = | 0 | ; | size of pixels (degrees) |
map_flagged_data | = | True | ; | flagged data put in a separated map |
file_format | = | 1 | ; | SANEPIC 0, HIPE 1 |
project_gaps | = | True | ; | gaps are projected to a pixel in the map, if so gap filling of noise only is performed iteratively. |
mask_file | = | mask.fits | ; | mask (fits file) : use to remove cross constraint due to strong sources |
lon | = | 0 | ; | LON nominal of the input scan data |
lat | = | 0 | ; | LAT nominal of the input scan data |
proj_code | = | TAN | ; | projection type, see wcslib projection code (default: TAN) |
axis_type | = | EQ | ; | axis type, EQ|GAL, (default EQ) |
eq2gal | = | False | ; | Convert J2000.0 EQ to GAL |
gal2eq | = | False | ; | Convert GAL to J2000.0 |
It is also possible to give a mask file as input, SANEPIC will then use it to remove crossing constrains on non-zero pixels and use its World Coordinate System header to construct the final map.
This is mainly a path and a suffix to the noise power spectra
[saneInv] | ||||
noise_dir | = | ./inputDir/ | ; | cov matrix directory |
cov_matrix_file | = | ; | this file contains the matrix you want to invert | |
cov_matrix_suffix | = | _psd.fits | ; | this file contains the matrix you want to invert |
All parameters related to the pre-processing of the data is summarized here, including the sampling frequency, order of the polynomial baseline, the cut-off frequency of the high-pass butherworth filter, the threshold frequency to be applied on the noise power spectra.
[saneProc] | ||||
sampling_frequency | = | 0 | ; | detectors sampling frequency [Hz] |
filter_frequency | = | 0 | ; | frequency of the high pass filter applied to the data [Hz] |
apodize_Nsamples | = | 100 | ; | number of samples to apodize |
fcut_file | = | fcutFile.fcut | ; | filename containing the frequency at which noise power spectra are thresholded |
poly_order | = | 0 | ; | baseline polynomia order (default : 0 no baseline) |
linear_baseline | = | True | ; | simple linear baseline removed from the data (False : default) |
correlation | = | True | ; | Set this keyword to False if correlations between detectors are not included in the analysis (True : default) |
fill_gap | = | True | ; | Do we fill the gaps ? (True : default) |
Most of the key parameter are set in [saneProc], so here are only the maximum number of iteration, a possible dumping of results during processing and the filename prefix of the maps
[sanePic] | ||||
iterW | = | 10 | ; | Write temporary map files on disk every iterW number of loop |
iterMAX | = | 2000 | ; | Maximum number of conjugate gradient loops (default: 2000) |
map_prefix | = | optimMap | ; | prefix for the fits file (default : optimMap) |
Input Files
A few additional files are needed in order to run SANEPIC
This file lists the input fits data files, one per line. An optionnal column can be added to assign file to processor/rank when using the parallelization per frame (note that the fits file can be compressed).
OD162_0x500022F9L_SpirePacsParallel_polaris_notdf_all_sanepic.fits
OD162_0x500022FAL_SpirePacsParallel_polaris_notdf_all_sanepic.fits.gz
This file lists the bolometers used in the SANEPIC processing, this can be a subset of the bolometer present in the data files. You can define either a common list for all data file (bolo_global_file), or a different list for each data file following the *_suffix mechanism. In this case your lists should be named file basename+bolo_suffix, i.e. if your data file is named fileA.fits, then the corresponding list of bolometer file will be named fileA.bolo.
These files contain the auto- and cross- power spectra of the noise. Thus for bolometer $i$ and $j$, the cross-power spectra between pure noise timeline (in Fourier space) $d_i$ and $d_j$ is defined as (see Patanchon et al. 2008)
$P_{ij}(w) = \frac{1}{n_{q}} \sum_{w_{min}(q)}^{w_{max}(q)}{\tilde{d}_{iw}^* \tilde{d}_{jw}}$
The fits file must contains 3 extensions, channels, equivalent to the table found in the data fits file, frequency a 1D image of size nFreq+1 listing the bins edge in frequency for the power spectra and finally PowerSpectra, which, for simplicity, contains all auto and cross power spectra, stored in a single 2D image of size nFreq x (nBolos x nBolos)
Thus, a typical (1.2 MB) Noise Noise Power Spectra file, describing nBolos=41 with nFreq=99 will look like :
Primary header |
7 records No data |
|
---|---|---|
Extension 1 | channels |
Header : 14 records Binary Table ( 5 x 41 ) |
Extension 2 | Frequency |
Header : 11 records IMAGE ( 100 ) |
Extension 3 | PowerSpectra |
Header : 11 records IMAGE ( 99 x 1681 ) |
This file list the threshold frequencies (in Hz), applied to the noise-noise power spectra, for each input fits data files or frames. It must follows the same order. Thus a typical file would read as :
0.000366 ; for the OD162_0x500022F9L_SpirePacsParallel_polaris_notdf_all_sanepic.fits
0.000366 ; for the OD162_0x500022FAL_SpirePacsParallel_polaris_notdf_all_sanepic.fits.gz file
Output Maps
The output maps are multi-extension fits file containing :
- Image, a 2D Image containing the final maximal likelihood map,
- Error, the error map corresponding to the previous Image, containing only the diagonal term of the noise-noise covariance matrix,
- Coverage, containing the number of hits within a sky pixel,
- Mask, a copy of the input mask, if any,
- IniFile, a copy of the input ini files, and,
- InputFiles, a extension which summarize the input files