PIMA Python wrappers

Date of last modification: 2017.08.31_11:13:57   

PIMA was designed to be "scriptable", i.e. called from another programs. The interface: control file that have to define all parameters, task and command line arguments that override the keywords defined in the control file is somewhat heavy weight. PIMA distribution provides 4 scripts and users are encouraged to develop their own. These scripts are also called wrappers, because they wrap calls to various PIMA tasks.

Contents:

The wrappers do not provide new functionality that PIMA does not have. They significantly simplify the user interface by expense of reducing functionality and by imposing rules on file names. If you need full functionality, for instance for processing unusual experiment, PIMA wrappers will not work for you. But the use of wrappers may be sufficient for processing many experiments.

Wrappers assume the file names obey the following convention:

The following wrappers are provided:

pf.py — a general PIMA wrapper.

 

usage: pf.py 
             [-h] [--version] [-v value] [-r] [-s] [-H]
             exp band
             {load,logs,gean,coarse,bpas,fine,mkdb,mktxt,splt,gain,autm,pict,map,sres,gaco,opal}
             [opts]
The wrapper has general options followed by three mandatory positional arguments followed by task specific optional arguments. General options that follow immediately after the wrapper name and before positional arguments:

There are three mandatory arguments:

Supported tasks:

pt.py — trial fringe fitting.

This wrapper performs fringe fit for an observation with given index. Fringe results are written in /tmp/1.fri and fringe residuals are written in /tmp/1.frr overriding keywords FRINGE_FILE and FRIRES_FILE. The task does not purge these files, and therefore, it appends results to their end. Task pt.py shows an 1D plot of residual phases and amplitudes versus frequency and a similar plot of residuals versus time. Examining fringe plots is the main function of this task. Task DEBUG_LEVEL: 6, and therefore, it prints verbose report about fringe fitting.

 

usage: pt.py 
             [-h] [--version] [-v value] [-r] [-s] [-H]
             exp band obs
             [opts]
Wrapper has general options followed by three mandatory positional arguments followed by PIMA options that are pairs keyword: value. There are three mandatory arguments:

pr.py — re-fringe VLBI experiment.

Task pr.py implements interface Solve &arrow; PIMA. It 1) analyzes Solve residual file, 2) finds observations that have been suppressed, computes predicted path delay on the basis of a priori path delay and adjustments from the Solve solution and computes correction to the a priori path delay with respect to the model used by the correlator; 3) generates a command file that calls PIMA with parameters of the search window centered with respect to the updated a priori path delay and with the specified window semi-width; 4) executes this command file; 5) creates databases in GVF format; 6) updates automatic suppression status.

There are several situations when PIMA re-fringe procedure helps to improve results:

  1. There was a strong RFI and fringe fitting procedure picked up fringes from the FRI;

  2. A priori delay rate was low and fringe fitting procedure picked up fringes from the phase calibration signal.

  3. There was a significant phase distortion in IFs after applying measured phase calibration and bandpass calibration. As a result, the amplitude of the secondary maximum of 2D Fourier transform that in the absence of phase distortion would be below the amplitude of the main maximum became higher than the level of the main maximum.

  4. A priori source position (and in a case of RadioAstron, a priori Space Radio Telescope position) was significantly (more than 1 mas) improved. Significant errors in a priori source position may result in quadratic term of fringe phase versus time.

  5. A source has marginal SNR (typically in range 4.5–6.5), and the thermal noise reduced the main maximum and increased a secondary maximum above the amplitude of the main maximum.

VTD/Post-Solve interactive solution should be made first. Option Print residu(A)ls: ON should be turn on, the spool file with solution listing be rewound, and residuals be generated (command Q). The the spool file with residuals should be copied into file VVVVVV/EEE/EEE_B_init.spl. An analyst should be check carefully the residual file. In particular, an analyst should check a) the experiment name and band name :-); b) whether solution is correct (wrms of residuals is close to what it is supposed to be); c) the spool file contains residuals. Residual section starts after line Residuals from Solve Symbols > at the 8-th position marks suppressed observation. All suppressed observations will be re-fringed.

There is a way to change the list of observations that will be re-fringed. If an analyst does not want to re-fringe some observations, the lines that correspond to these observations should be either removed from the listing file or the character at the 8th column of the rows that correspond to those observations should be changed to K. Alternatively, if an analyst would like to re-fringe a given observation even it it is not suppress, the character at 8th column should be changed to R. Re-fringing an observation of a source that had a priori position errors exceeding 1 arcsec may improve the SNR. 

Usage: pr.py 
             [-h] [--version] [-v value] [-r] [-s] [-H]
             exp band snr
             [opts]
Wrapper has general options followed by three mandatory positional arguments followed by PIMA options that are pairs keyword: value. There are three mandatory arguments:

The wrapper creates log file VVVVVV/EEE/EEE_samb.log.

pu.py — update suppression status after re-fringing.

VTD/Post-Solve supports so-called automatic suppression status. This status depends on a number of factors including detection status and other parameters. When PIMA creates a GVF database it sets this status for version 1. But the status does not automatically propagate to version 2 and higher. Wrapper pu.py propagates the status from version 1 database to the higher version.

Let us consider the following situation. A given observations had SNR 4.7 and therefore was treated as unconditionally suppressed. VTD/post-Solve does not show such observation and does not allow to restore it. After re-fringing then SNR grew to 6.8, i.e. the observations was detected. Task mkdb created GVF database version 1. The suppression status is version dependent. The observation is marked as good in version 1, but unconditionally suppressed in version 2.

Wrapper pu.py will set status "suppressed, but recovered" in version 2. Then a user can reset status to "good". 

usage: pu.py 
             [-h] [--version] [-v value] [-r] [-s] [-H]
             exp

automap.py — Automatic imaging of a given visibility file.

This wrapper calls DIFMAP and preforms automatic imaging using script pima_mupet_01.dfm developed by Martin Shepherd and Greg Taylor. 
usage: automap.py 
             [--version]
             uva_file
The input for the wrapper is the file with averaged calibrated visibilities in FITS-IDI format.

The wrapper assumes the filename with averaged calibrated visibilities has the following form SSSSSS/EEE_uvs/JJJJJJJJJJ_B_uva.fits where SSSSSS is the directory specified in the keyword SESS_CODE of PIMA control file. The wrapper generates 5 output files: SSSSSS/EEE_uvs/JJJJJJJJJJ_B_map.fits — FITS image, SSSSSS/EEE_uvs/JJJJJJJJJJ_B_uvs.fits — self-calibrated, scan averaged visibilities in FITS format, SSSSSS/EEE_uvs/JJJJJJJJJJ_B.mod — ascii file with Clean components of the image, SSSSSS/EEE_uvs/JJJJJJJJJJ_B.win — coordinates of four corners of CLEAN windows used be the imaging process, SSSSSS/EEE_uvs/JJJJJJJJJJ_B.par — command file created by the DIFMAP.

The quality of automatic image may or may not be satisfactory. Automatic image does not perform flagging. If the visibility data are either too high or too low for some IFs due to errors in amplitude calibration, or a portion of data has garbage visibilities at some station(s) because the antenna(s) were not on source, the quality of automatic image will be disappointing at best, or totally garbage at worst. Running onof and gaco tasks usually solve these problems and substantially reduces the chances that the automatically generated images will have unsatisfactory quality.

In general, analyst should scrutinize automatically generated images and decide whether to keep them or re-image them manually.

imadir.py — Automatic imaging for all files with averaged visibilities in a given directory. 

usage: imadir.py 
             [-h] [--version] [-pict] [-H]
             directory
This wrapper scans the specified directory, searches file with ending uva.fits or uvm.fits and executes wrapper automap.py for each file, i.e. generates automatically the image and picture files

This document was prepared by Leonid Petrov
Last update: 2020.05.02