Documentation

SANEPIC Documentation

This page present quickly the SANEPIC Workflow and well as its input data and files format.

Workflow

SANEPIC is organized in several programs to be used in a comprehensive framework :

  • sanePre : is used to perform preprocessing steps.
  • sanePos : will take care of all computations with regards to astronomical positions
  • saneInv : will inverse the noise-noise power spectra (see Patanchon et al. 2008)
  • sanePic : is the main program where the complete inversion of the map-making equation is performed.

All the different programs use a central configuration file and some other input files. The workflow of SANEPIC is summarized with this graphic :

 

sanePre
This program ensures that all directories and files are present and will extract the data from the fits files into internal directories for future faster access. On clusters it will also distribute the data among the different nodes. This program is only launched once at the begninning of the processing, as long as the data/position/flag are unchanged.
sanePos
This program determines the map parameters and size and compute the pixel index of all bolometers within this map, properly handling the flagged data into separate maps (if needed). It can also perfom coordinate transformation between J2000 Equatorial and Galactic system.
sanePic
This program will process data written by sanePre, saneProc and saneInv to compute the maximum likelihood solution of the map-making equation. It is performed iteratively, using a conjugate gradient approach.

The input data format is described below, a well as the output maps files.


Data Format

For the time ordered data, SANEPIC use a simple, multi-extension, fits file. Each fits file, or frame, describe a continuous set of data as :

  • signal & mask are two mandatory 2D images, with dimensions nTime x nBolo, coded in 64 and 8 bits respectively.
  • the positions of the bolometers on the sky, is also mandatory and can be provided in two different ways :
  • channels, a binary table containing at least one columns, names, and nBolo lines naming each bolometers.
  • time, a 1D image, with nTime items, can also be provided, in order to perform some checks.
signal & mask
The signal table contains the calibrated signal in unit of the final map. It can include glitches and other artefacts as long as their are masked. The mask table should include all non physical effect. The convention used for the mask is 0 for good data, any other value would be treated as bad data.
lon & lat
The lon and lat contains either the right ascension and declinaison or the galactic longitude and latitude, of all bolometers at all timestamps, following the ordering of the signal and mask images. They must be in degree. This format is referred as the HIPE format in the ini file below.
reference position & offsets
The offsets table consists of 3 columns, names, dX, dY, and nBolo lines, describing the relative offsets of the bolometers. The reference position contains 3 rows, lon, lat & phi, and nTime lines, describing the position on the sky of the reference offset and the position angle between the bolometers and sky plane. The offsets should be provided using a TAN projection. This format is referred as the SANEPIC format in the ini file
channels
The channel table has the same order as the nBolo dimension in the signal and mask images. This is only an indexing of the channels and will be used later to select group of bolometers to produce maps with SANEPIC, for e.g. jackknife maps.
primary header
Some keyword are searched in the primary header, namely EQUINOX, DATE-OBS, RADESYS, RESTWAV, RESTFRQ, TIMESYS, CREATOR, INSTRUME, OBJECT, TELESCOP, OBSERVER, UNIT. This keyword will be propagated into the final fits file.

Thus, a typical nTime=152612 and nBolo=41 dataset (108 MB) would look like :

Primary header   132 records
No data
Extension 1 signal Header : 25 records
IMAGE ( 152612 x 41 )
Extension 2 lon Header : 27 records
IMAGE ( 152612 x 41 )
Extension 3 lat Header : 27 records
IMAGE ( 152612 x 41 )
Extension 4 mask Header : 19 records
IMAGE ( 152612 x 41 )
Extension 5 time Header : 24 records
IMAGE ( 152612 )
Extension 6 channels Header : 52 records
Binary Table ( 6 x 41 )
Extension 7 reference position Header : 52 records
Binary Table ( 24 x 152612 )
Extension 8 offsets Header : 43 records
Binary Table ( 22 x 41 )

Note that the extensions are accessed by their names (EXTNAM keyword), and not by their position/number in the fits file.


Ini Files

The input file of SANEPIC follow the ini format. It consists of several sections describing the input of each programs : (complete example file)

[common]

The common section is used to describe the fields used by all programs, mainly input output and temporary directories, lists of fits files and bolometers

[common]        
data_directory = ./data/ ; source data directory
input_directory = ./inputDir/ ; input directory with all the configurations files
output_dir = ./Output/ ; output directory
temp_dir = ./tempDir/ ; temporary directory
fits_filelist = listFile.list ; file containing fits file names
bolo_global_file =   ; every scans have the same detector list
bolo_suffix = .bolo ; bolometers filelist suffix

The format of the input files and an explanation of the suffix mechanism will follow

[sanePos]

All quantities related to positions and map will be here, pixel size, projection center, projection type, coordinate type, coordinates transformation (galactic/equatorial), file format, masks and mapping of flagged data

[sanePos]        
pixsize = 0 ; size of pixels (degrees)
map_flagged_data = True ; flagged data put in a separated map
file_format = 1 ; SANEPIC 0, HIPE 1
project_gaps = True ; gaps are projected to a pixel in the map, if so gap filling of noise only is performed iteratively.
mask_file = mask.fits ; mask (fits file) : use to remove cross constraint due to strong sources
lon = 0 ; LON nominal of the input scan data
lat = 0 ; LAT nominal of the input scan data
proj_code = TAN ; projection type, see wcslib projection code (default: TAN)
axis_type = EQ ; axis type, EQ|GAL, (default EQ)
eq2gal = False ; Convert J2000.0 EQ to GAL
gal2eq = False ; Convert GAL to J2000.0

It is also possible to give a mask file as input, SANEPIC will then use it to remove crossing constrains on non-zero pixels and use its World Coordinate System header to construct the final map.

[saneInv]

This is mainly a path and a suffix to the noise power spectra

[saneInv]        
noise_dir = ./inputDir/ ; cov matrix directory
cov_matrix_file =   ; this file contains the matrix you want to invert
cov_matrix_suffix = _psd.fits ; this file contains the matrix you want to invert
[saneProc]

All parameters related to the pre-processing of the data is summarized here, including the sampling frequency, order of the polynomial baseline, the cut-off frequency of the high-pass butherworth filter, the threshold frequency to be applied on the noise power spectra.

[saneProc]        
sampling_frequency = 0 ; detectors sampling frequency [Hz]
filter_frequency = 0 ; frequency of the high pass filter applied to the data [Hz]
apodize_Nsamples = 100 ; number of samples to apodize
fcut_file = fcutFile.fcut ; filename containing the frequency at which noise power spectra are thresholded
poly_order = 0 ; baseline polynomia order (default : 0 no baseline)
linear_baseline = True ; simple linear baseline removed from the data (False : default)
correlation = True ; Set this keyword to False if correlations between detectors are not included in the analysis (True : default)
fill_gap = True ; Do we fill the gaps ? (True : default)
[sanePic]

Most of the key parameter are set in [saneProc], so here are only the maximum number of iteration, a possible dumping of results during processing and the filename prefix of the maps

[sanePic]        
iterW = 10 ; Write temporary map files on disk every iterW number of loop
iterMAX = 2000 ; Maximum number of conjugate gradient loops (default: 2000)
map_prefix = optimMap ; prefix for the fits file (default : optimMap)

Input Files

A few additional files are needed in order to run SANEPIC

fits filelist

This file lists the input fits data files, one per line. An optionnal column can be added to assign file to processor/rank when using the parallelization per frame (note that the fits file can be compressed).

# example of listFile.list
OD162_0x500022F9L_SpirePacsParallel_polaris_notdf_all_sanepic.fits
OD162_0x500022FAL_SpirePacsParallel_polaris_notdf_all_sanepic.fits.gz
bolometer list

This file lists the bolometers used in the SANEPIC processing, this can be a subset of the bolometer present in the data files. You can define either a common list for all data file (bolo_global_file), or a different list for each data file following the *_suffix mechanism. In this case your lists should be named file basename+bolo_suffix, i.e. if your data file is named fileA.fits, then the corresponding list of bolometer file will be named fileA.bolo.

# example of list.bolo
Bolo1
Bolo2
#Bolo3
Bolo4
Noise-Noise Power Spectra

These files contain the auto- and cross- power spectra of the noise. Thus for bolometer $i$ and $j$, the cross-power spectra between pure noise timeline (in Fourier space) $d_i$ and $d_j$ is defined as (see Patanchon et al. 2008)

$P_{ij}(w) = \frac{1}{n_{q}} \sum_{w_{min}(q)}^{w_{max}(q)}{\tilde{d}_{iw}^* \tilde{d}_{jw}}$

The fits file must contains 3 extensions, channels, equivalent to the table found in the data fits file, frequency a 1D image of size nFreq+1 listing the bins edge in frequency for the power spectra and finally PowerSpectra, which, for simplicity, contains all auto and cross power spectra, stored in a single 2D image of size nFreq x (nBolos x nBolos)

Thus, a typical (1.2 MB) Noise Noise Power Spectra file, describing nBolos=41 with nFreq=99 will look like :

Primary header   7 records
No data
Extension 1 channels Header : 14 records
Binary Table ( 5 x 41 )
Extension 2 Frequency Header : 11 records
IMAGE ( 100 )
Extension 3 PowerSpectra Header : 11 records
IMAGE ( 99 x 1681 )
fcut file

This file list the threshold frequencies (in Hz), applied to the noise-noise power spectra, for each input fits data files or frames. It must follows the same order. Thus a typical file would read as :

# example of fCutFile.fcut
0.000366 ; for the OD162_0x500022F9L_SpirePacsParallel_polaris_notdf_all_sanepic.fits
0.000366 ; for the OD162_0x500022FAL_SpirePacsParallel_polaris_notdf_all_sanepic.fits.gz file

Output Maps

The output maps are multi-extension fits file containing :

  • Image, a 2D Image containing the final maximal likelihood map,
  • Error, the error map corresponding to the previous Image, containing only the diagonal term of the noise-noise covariance matrix,
  • Coverage, containing the number of hits within a sky pixel,
  • Mask, a copy of the input mask, if any,
  • IniFile, a copy of the input ini files, and,
  • InputFiles, a extension which summarize the input files