IMRED module

imred is a module that implements some tools for basic data reduction: overscan subraction, bias subtraction, dark subtraction, and flat fielding. Routines are included for basic combination of images that can be used to make basic calibration products. Data are read and processed using pyvista Data structures (a superset of astropy CCDData/NDData, see Dataclass module), which carry along both the image data and associated uncertainty and mask arrays.

The routines are implemented with the Reducer class, which performs various tasks: reads data from disk and does various reduction steps, either individually or as a group. Also contains methods for constructing calibration frames

Reducer

Given information about instrument detector, a reducer object reads and, as requested, reduces images. A class is instantiated for a given instrument. Necessary information can be specified on instantiation, but generally is loaded given an input instrument name, e.g., ARCTIC, ARCES, DIS, TSPEC. In this case, configuration is read from a configuration file called {inst}_config.yml; pyvista contains several of these in the PYVISTA_DIR/python/pyvista/data directory tree. For a new instrument, they should be straightforward to construct, see below.

For a given instrument, there can be multiple files associated with each exposure, e.g. DIS blue/red, and the configuration file then specifies the multiple configurations. During reduction operations, all files for an exposure (i.e., multiple channels) are processed, with data returned as a list of pyvista Data objects.

The reading function of the Reducer allows for FITS files to be read by specifying a sequence number (rather than typing the full name), if the data taking system supports unique identification by sequence number. File names are constructed using a template: {dir}/{root}{formstr}.*, where {dir}, {root}, and {formstr} are customizable attributes of the Reducer. For example, if {dir} is defined as some input directory, {root} is ‘*’, and {formstr} is {:04d}, then if you were to, e.g. rd(28), it would look for a file {dir}/0028.fits’ and read it. You could also specify rd(‘Image0028.fits’), in which case it would read from {dir}/Image0028.fits, or you could specify rd(‘./Image0028.fits’) in which case it would read from the current working directory.

The basic reading method of the reducer is rd(); however, this is generally not the preferred method, since it does not do any processing (e.g. overscan subtraction), so the noise cannot be correctly calculated. Instead, the reduce() method is preferred, which will read and also perform basic reduction depending on what calibration frames are specified as optional arguments.

The reduce() method will read and perform some reductions. It will perform overscan subtraction if self.biastype has been set >=0, It will perform bias frame subtraction if a bias frame is passed with the bias= keyword, dark subtraction if a dark frame is passed with the dark= keyword, and flat fielding if a flat frame is passed with the flat= keyword; see below for construction of calibration frames. There is also the possibility of scattered light subtraction (e.g., for a multi-order spectrograph) if a scattered light parameter has been set, cosmic ray rejection if a cosmic ray parameter is passed with crbox=, and bad pixel masking if a bad pixel mask has been set.

Calibration frames can be created using the mkbias(), mkdark(), mkflat(), and mkspecflat() methods. These take lists of frames as arguments and combine the input frame using a median filter (with normalization for flats). The calibration frames are also returned as pyvista Data objects, so that the uncertainties in the calibration products can be assessed, and propagated.

Processing can be done with associated description and image display of the steps being taken. If the verbose attribute is set to True, more text is displayed. If the display attribute is set to a pyvista TV object, then frames will be displayed as they are processed, and user input is requested to proceed from step to step, to allow the user to look at the frames.

Processed data can be written to FITS file(s) on disk using the write() method, which is just a wrapper to the pyvista Data object write() method, but allows for multiple channels in an instrument (which would be written to multiple files).

Processed data should be read using the pyvista Data.read() method; the Reducer object would NOT be used to read processed data!

Attributes:

• dir : default directory to read images from (can be overridden)

• root : default root file name (can be overridden)

• formstr : format string used to search for images using sequence number

• verbose : turns on verbose output

• gain : gain to use to calculate initial uncertainty

• rn : readout noise used to calculate initial uncertainty

• biastype : specifies algorithm for bias subtraction

• biasbox : pyvista BOX object giving region(s) to use for overscan

• trimbox : pyvista BOX object giving region(s) to trim reduced image to, if requested

• normbox : pyvista BOX object giving region(s) to use for normalization of flats

Methods (see below for full description with keywords) :

• rd: reads image from disk. For convenience, files can be identified by a sequence number alone, and the input directory will be searched for files matching the input format ({dir}/{root}{formstr}.fits*) with the input file number

• overscan : subtracts overscan, and calculates uncertainty

• bias : subtracts input combined bias frame

• flat : divides by input combined flat frame

• reducereads, overscan subtracts (per biastype), bias subtracts if

  given a combined bias, flat fields if given a combined flat, etc.

• getcube: reads a set of frames into a data cube

• median: median filters images in cube

• sum: : sums images in cube

• mkbias: given input bias frames, creates combined bias with median combination after overscan subtraction

• mkflat: given input flat field frames, creates combined bias with median combination with normalization, after overscan subtraction, and bias subtraction if given combined bias

• mkspecflat: creates flat field as above, but then removes structure along columns using spectral signature calculated from a boxcar filter of a central set of rows, to produce a flat field with spectral signature removed

• write : writes out Data object(s) to fits file(s) (more than one if multiple channels), using pyvista dataclass.Data.write()

Making a new instrument configuration file

Configuration files are made using the YAML format, e.g.:

---
channels : ['blue','red']
formstr  : ['{:04d}b','{:04d}r']
gain     : [1.68,1.88]
rn       : [4.9,4.6]
crbox    : [1,11]
biastype : 0
biasregion :
   - [[0,2098],[0,1078]]
   - [[0,2098],[0,1078]]
biasbox  : 
  - [[2050,2096],[0,1023]]   # xs, xe, ys, ye
  - [[2050,2096],[0,1023]]
trimbox  : 
  - [[0,2047],[0,1023]]
  - [[0,2047],[0,1023]]
normbox  : 
  - [[1000,1050],[500,600]]
  - [[1000,1050],[500,600]]

imred functions

class pyvista.imred.Reducer(inst=None, conf='', dir='./', root='*', formstr='{:04d}.f*', gain=1, rn=0.0, saturation=4294967296, verbose=True, nfowler=1, trim=False)

Class for reducing images of a given instrument

Variables

• dir (str) –

• root (str) –

• verbose (bool) –

• inst (str) –

• badpix (str) –

• scat (int) –

• mask (str) –

• transpose (bool) –

• scale (float) –

• biastype (int) –

• gain (float) –

• rn (float) –

• namp (int) –

• crbox (list) –

• biasavg (int) –

• biasregion –

• trimbox –

• outbox –

log(files=None, htmlfile=None, ext=None, hdu=0, channel='', cols=None, display=None, hexdump=False)

Create chronological image log from file headers in default

directory.

If any .csv file exists in the directory, its contents are added as a table to htmlfile

Parameters

• htmlfile (str, default=None) – if specified, write HTML log to htmlfile

• ext (override default extension to search) –

• cols (array-like, str, specifies which FITS header cards to output,) – default=None, which will use cards as defined in the Reducer object, if they have been set by configuration file, or, otherwise will use [‘DATE-OBS’,’OBJNAME’,’RA’,’DEC’,’EXPTIME’]

• display (if not None, specifies tv tool in which to) – display each image and make png thumbnail, to include in htmlfile

Return type

astropy table from FITS headers

movie(ims, display=None, out='movie.mp4', channel=0, min=None, max=None, box=None, text=True, fps=None)

Create animated gif of images

Parameters

• ims (list of image numbers.) –

• display (pyvista TV object) –

• out (output file name) –

reduce(num, channel=None, ext=0, crbox=None, crsig=5, objlim=5, sigfrac=0.3, bias=None, dark=None, flat=None, scat=None, scat_smooth=None, badpix=None, solve=False, return_list=False, display=None, trim=True, seeing=2, utr=False)

Reads data from disk, and performs reduction steps

as determined from command line parameters

Parameters

• id (int or str) – Number or string specifying file to read. If a number, the filename will be constructed based on dir and formstr attributed of Reducer object. Without any additional command-line arguments, data will be read, overscan subtracted, and uncertainty array populated based on gain and readout noise in Reducer attributes

• display (TV object, default=None) – if specified, pyvista TV object to display data in as various reduction steps are taken

• channel (int, default= None) – if specified, channel to reduce if instrument is multi-channel (multi-file), otherwise all channels will be read/reduced

• bias (Data object, default= None) – if specified, superbias frame to subtract

• dark (Data object, default= None) – if specified, superdark frame to subtract

• flat (Data object, default= None) – if specified, superflat frame to divide by

• crbox (list or str, default=None) – if specified, parameter to pass to CR rejection routine, either 2-element list giving shape of box for median filter, or ‘lacosmic’

• scat (integer, default=None) – if specified, do scattered light correction, getting estimate every scat pixels

• badpix (int, default=None) – if specified, set masked pixels to specified value

• trim (bool, default=True) – trim image after calibration, irrelevant if red.trimg=True

• solve (bool, default=False) – attempt to plate-solve image after reduction, requires local astrometry.net

• seeing (float, default=2) – seeing used to find stars if solve=True

rd(num, ext=0, dark=None, channel=None, utr=False)

Read an image

Parameters

int) (num (str or) –

Returns

image (Data )

Return type

Data object, but noise will be incorrect without overscan subtraction

overscan(im, display=None, channel=None)

Overscan subtraction

trimimage(inim, trimimage=False)

Trim image by masking non-trimmed pixels May need to preserve image size to match reference/calibration frames, etc.

boxfromheader(im, nx, ny)

Set boxes from image header

bias(im, superbias=None)

Superbias subtraction

dark(im, superdark=None)

Superdark subtraction

flat(im, superflat=None, display=None)

Flat fielding

scatter(inim, scat=None, display=None, smooth=None, smooth2d=31, transpose=False)

Removal of scattered light (for multi-order/object spectrograph)

Remove scattered light by looking for valleys in cross-sections across traces and fitting a 2D surface to the low points. Cross-sections are smoothed before finding the valleys, and interpolated surface is also smoothed. Some attempt is made to reject outliers before fitting the final surface, which is subtracted from the image.

Parameters

• im (Data object) – input image to correct

• transpose (bool, default=False) – set to true if spectra run along columns

• scat (integer, default=None) – get scattered light measurements every scat pixels. If None, no correction

• smooth (integer, default=3) – boxcar width for smoothing profile perpendicular to traces before looking for valleys

• display (TV object, default=None) – if set, show the scattered light measurements

crrej(im, crbox=None, crsig=5, display=None, objlim=5.0, sigfrac=0.3, fsmode='median', inbkg=None)

Cosmic ray rejection using spatial median filter or lacosmic.

If crbox is given as a 2-element list, then a box of this shape is run over the image. At each location, the median in the box is determined. For each pixel in the box, if the value is larger than crsig*uncertainty (where uncertainty is taken from the input.uncertainty.array), the pixel is replaced by the median. If crsig is a list, then multiple passes are done with successive values of crsig (which should then be decreasing), but only neighbors of previously flagged CRs are tested. Affected pixels are flagged in input.bitmask

If crbox=’lacosmic’, the LA Cosmic routine, as implemented in astroscrappy is run on the image, with default options, but objlim, fsmode, and inbkg can be specified.

Parameters

• crbox (list, int shape of box to use for median filters, or 'lacosmic') –

• crsig (list/float, default 5, threshold for CR rejection if using spatial) – median filter; if list, do multiple passes, with all passes after first only on neighbors of previously flagged CRs

• objlim (for LAcosmic, default=5) –

• fsmod (for LAcosmic, default='median') –

• inbkg (for LAcosmic, default=None) –

• display (None for no display, pyvista TV object to display) –

badpix_fix(im, val=0.0)

Replace bad pixels

platesolve(im, scale=0.46, seeing=2, display=None, thresh=10)

try to get plate solution with astrometry.net

noise(pairs, rows=None, cols=None, nbox=200, display=None, channel=None, levels=None, skip=1)

Noise characterization from image pairs

display(display, id)

Reduce and display image

write(im, name, overwrite=True, png=False, wave=None)

write out image, deal with multiple channels

getcube(ims, **kwargs)

Read images into data cube

sum(ims, return_list=False, **kwargs)

Coadd input images

combine(ims, normalize=False, display=None, div=True, return_list=False, type='median', sigreject=5, **kwargs)

Combine images from list of images

mkbias(ims, display=None, scat=None, type='median', sigreject=5, trim=False)

Driver for superbias combination (no superbias subtraction no normalization)

ims : list of frames to combine display : TV object, default= None

if specified, displays bias and individual frames-bias for inspection

typestr, default=’median’

combine method

sigrejectfloat

rejection threshold for combine type=’reject’, otherwise ignored

mkdark(ims, ext=0, bias=None, display=None, scat=None, trim=False, type='median', sigreject=5, clip=None)

Driver for superdark combination (no normalization)

Parameters

• ims (list of frames to combine) –

• display (TV object, default= None) – if specified, displays dark and individual frames-dark for inspection

• bias (Data object, default=None) – if specified, superbias to subtract before combining darks

• type (str, default='median') – combine method

• sigreject (float) – rejection threshold for combine type=’reject’, otherwise ignored

• clip (float, default=None) – if specified, set all values in output dark < clip*uncertainty to zero in master dark

mkflat(ims, bias=None, dark=None, scat=None, display=None, trim=False, ext=0, type='median', sigreject=5, spec=False, width=101, littrow=False, normalize=True, snmin=50, clip=None)

Driver for superflat combination, with superbias if specified, normalize to normbox

Parameters

• ims (list of frames to combine) –

• display (TV object, default= None) – if specified, displays flat and individual frames/flat for inspection

• bias (Data object, default=None) – if specified, superbias to subtract before combining flats

• dark (Data object, default=None) – if specified, superdark to subtract before combining flats

• scat –

• type (str, default='median') – combine method

• sigreject (float) – rejection threshold for combine type=’reject’, otherwise ignored

• spec (bool, default=False) – if True, creates “spectral” flat by taking out wavelength shape

• littrow (bool, default=False) – if True, attempts to fit and remove Littrow ghost from flat, LITTROW_GHOST bit must be set in bitmask first to identify ghost location. Ignored if spec==False

• width (int, default=101) – window width for removing spectral shape for spec=True

Return type

Data object with combined flat

mkspecflat(flats, width=101, display=None, littrow=False, snmin=50)

Spectral flat takes out variation along wavelength direction

class pyvista.imred.DET(inst=None, gain=0.0, rn=0.0, biastype=0, biasbox=None, normbox=None, trimbox=None, formstr='{:04d}')

Defines detector class

pyvista.imred.look(tv, pause=True, files=None, list=None, min=None, max=None)

Displays series of files

pyvista.imred.getfiles(type, listfile=None, filter=None, verbose=False)

Get all files of desired type from specified directory. If file is specified, read numbers from that file, else use IMAGETYP card

pyvista.imred.disp(tv, hd, min=None, max=None, sky=False)

Displays HDU or data array on specified ds9/tv device

pyvista.imred.getinput(text, display)

print text, get a key input from display