.. _programmers_notes:

.. |br| raw:: html

   <br />
   
.. index::
 single: Programmers notes
 
Programmers Notes
=====================

CSV tables
-----------

In some modules CSV tables may by imported (mk_master_cal).

The implemented style is:

.. csv-table::
 :delim: ,

 serial, chopper, f, a, p, stddev a, stddev p
 1, 1, 1.232800e-01, 2.000200e-01, 8.823600e+01, 2.734738e-04, 2.734738e-04
 1, 0, 1.000000e+00, 1.892500e-01, 1.112900e+02, 1.312173e-03, 1.312173e-03


.. code-block:: text

 serial, chopper, f, a, p, stddev a, stddev p
 1, 1, 1.232800e-01, 2.000200e-01, 8.823600e+01, 2.734738e-04, 2.734738e-04
 1, 0, 1.000000e+00, 1.892500e-01, 1.112900e+02, 1.312173e-03, 1.312173e-03

CVS tables are always TEXT. When performing actions based on tables, be aware that **1.0e+00** and **1.0E+00** are **NOT THE SAME** |br|
Also in case of mathematics a string must be converted into a number and back: |br|
.. 1.0e+00 -> double is ok, 1.0e+00 -> int is not ok; so internally we always go ``1.0e+00 -> double -> int`` (in case). |br|
In case of string data we always compare lower case! |br|
All frequencies you use have to be sorted from low to high.





.. _ats_header:

ATS header
-----------

ats stands for advanced timeseries. |br|
In languages such as C++ you do simlilar like (here with C++ & Qt) 

.. code-block:: c++

 ATSHeader_80   bin_atsheader;   //!< binaray header for read/write
 
 ....
 file.setFileName("myatsfile.ats");
 file.open(QIODevice::ReadOnly))
 QDataStream qds;
 qds.setDevice(file);
 qds.readRawData((char*) &bin_atsheader, sizeof(bin_atsheader));
 qDebug() << "samples: " << bin_atsheader.uiSamples;
 
In other languages reading a binary file is maybe similar.


.. code-block:: c++

    #ifndef ATSHEADER_80_DEF_H
    #define ATSHEADER_80_DEF_H

    #include <cstdint>

    //
    //
    //
    //         array of chars are not NULL terminated !
    //         achChanType   [2] = "Ex" is not NULL terminated
    //
    //

    #define C_ATS_CEA_NUM_HEADERS   1023     /*!< maximum number of 1023 headers of the ATS sliced file.*/


    /*!
    \brief The ATSSliceHeader_s struct
    A sliced ats file ALWAYS contains a default header and C_ATS_CEA_NUM_HEADERS (1023) = 1024 total.<br>
    The FIRST and DEFAULT slice contains ALL information (total samples of all slices)<br>
    The 0 ... 1022 slices (=  ATS_CEA_NUM_HEADERS ) contain the individual descriptions<br>
    start tsdata = 400h = 1024th byte<br>
    start of CEA data is 400h + 3FFh * 20h = 83E0h<br>
                            1024 + 1023 * 32  = 33760<br>

    assuming 3 slices of 4096 iSamples will be 12,288 samples in the header,
    followed by 3 slice headers with 4096 iSamples each
    and 1019 empty slice headers<br>
    the data has to sorted out on/after 83E0h according to the slice info

    */

    struct ATSSliceHeader_1080
    {
        std::uint32_t uiSamples;                    //!< 000h  number of samples for this slice
        std::uint32_t uiStartDateTime;              //!< 004h  startdate/time as UNIX timestamp (UTC)
        double        dbDCOffsetCorrValue;          //!< 008h  DC offset correction value in mV
        float         rPreGain;                     //!< 010h  originally used pre-gain (GainStage1) - LSB is the same for all slices
        float         rPostGain;                    //!< 014h  originally used post-gain (GainStage2) - LSB is the same for all slices
        std::int8_t   DCOffsetCorrOn;               //!< 018h  DC offset correction was on/off for this slice
        std::int8_t   reserved[7];                  //!< 020h  reserved bytes to get to word / dword boundary
    };

    /*! @todo TX info
    maybe UTM Zone number
    maybe UTM letter
    Tx times six double at least
    Tx base freq

    */

    /*!
    \brief The ATSComments_80 struct for ats header 80,81 and 90 <br>
    Some software will automatically convert to higher version<br>
    Some fields are external - the user has to set then; examples:<br>
    northing, easting, UTM Zone : these can be used for high density grids<br>
    */
    struct ATSComments_80 {
        char          achClient        [16];        //!< 000h  UTF-8 storage, used in EDI
        char          achContractor    [16];        //!< 010h  UTF-8 storage, used in EDI
        char          achArea          [16];        //!< 020h  UTF-8 storage, used in EDI
        char          achSurveyID      [16];        //!< 030h  UTF-8 storage, used in EDI
        char          achOperator      [16];        //!< 040h  UTF-8 storage, used in EDI
        char          achSiteName     [112];        //!< 050h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
        char          achXmlHeader     [64];        //!< 0C0h  UTF-8 storage  points to the corresponding XML file, containing addtional information for this header; no PATH=XML file is in the same directory as the ats file
        // deleted September 2018 char          achComments   [512];          //!< 100h  UTF-8 storage  comment block starts (comment field can start with "weather:" but has no meaning
        // new September 2018
        char          achComments     [288];        //!< 100h  UTF-8 storage  comment block starts (comment field can start with "weather:" but has no meaning
        char          achSiteNameRR   [112];        //!< 220h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
        char          achSiteNameEMAP [112];        //!< 290h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars); INDICATES a default EMAP center station, where we expect the H (magnetic)


    };


    /*!
    \brief The ATSHeader_80 struct<br>
    reference is lat and long; additional entries like northing and easting should be empty // not used<br>
    these entries can be overwritten by the user in case of high density grids <br>
    The site_name (site name, achSiteName, is in the comment field; the site_no, site_num, site_number, site number is nver stored; this is part of external numeration
    */
    struct ATSHeader_80 {
        std::uint16_t uiHeaderLength;               //!< 000h  length of header in bytes (default 1024 and  33760 for CEA)
        std::int16_t  siHeaderVers;                 //!< 002h  80 for ats, 81 for 64bit possible / metronix, 1080 for CEA / sliced header

        // This information can be found in the ChannelTS datastructure
        std::uint32_t uiSamples;                    //!< 004h  amount of samples (each is a 32bit / 64bit int INTEL byte order little endian) in the file total of all slices; if uiSamples == UINT32_MAX, uiSamples64bit at address 0F0h will be used instead; uiSamples64bit replaces uiSamples in that case; do not add!
        //!<
        float         rSampleFreq;                  //!< 008h  sampling frequency in Hz
        std::uint32_t uiStartDateTime;              //!< 00Ch  unix TIMESTAMP (some computers will revert that to negative numbers if this number is grater than 32bit signed); 2106-02-07T06:28:14 is the END of this format
        double        dblLSBMV;                     //!< 010h  least significant bit in mV ()
        std::int32_t  iGMTOffset;                   //!< 018h  not used, default 0; can be used as "UTC to GMT"
        float         rOrigSampleFreq;              //!< 01Ch  sampling frequency in Hz as ORIGINALLY recorded; this value should NOT change (for example after filtering)

        //The required data could probably found in the HardwareConfig
        std::uint16_t uiADUSerNum;                  //!< 020h  Serial number of the system (logger)
        std::uint16_t uiADCSerNum;                  //!< 022h  Serial number of the ADC board (ADB)
        std::uint8_t  uiChanNo;                     //!< 024h  Channel number
        std::uint8_t  uiChopper;                    //!< 025h  Chopper On (1) / Off (0); e.g. chopper is on for 512Hz and lower

        // Data from XML Job-specification
        char          achChanType   [2];            //!< 026h  (Ex, Ey, Ez, Hx, Hy, Hz, Jx, Jy, Jz, Px, Py, Pz, Rx, Ry, Rz and so on)
        char          achSensorType [6];            //!< 028h  (MFS06 MFS06e MFS07 MFS07e MFS10e SHFT02 SHF02e FGS02 FGS03 FGS03e etc. e.g. the "-" in MFS-06e is skipped)
        std::int16_t  siSensorSerNum;               //!< 02Eh  Serial number of connected sensor

        float         rPosX1;                       //!< 030h  e.g. South negative [m]
        float         rPosY1;                       //!< 034h  e.g. West negative [m]
        float         rPosZ1;                       //!< 038h  e.g. top, sky [m]
        float         rPosX2;                       //!< 03Ch  e.g. North positive [m]
        float         rPosY2;                       //!< 040h  e.g. East positive [m]
        float         rPosZ2;                       //!< 044h  e.g. bottom, earth [m]

        // not used any more use pos values!!; GUI interfaces using Length and direction MUST calculate pos x,y,z and set above.
        float         rDipLength;                   //!< 048h  e.g. to be calculated; should not be used - my be over written in FUTURE
        // not used any more use pos values!!
        float         rAngle;                       //!< 04Ch  e.g. to be calculated; should not be used - my be over written in FUTURE

        // Data from Selftest
        float         rProbeRes;                    //!< 050h  [ohm]
        float         rDCOffset;                    //!< 054h  [mV]
        float         rPreGain;                     //!< 058h  e.g. Gain Stage 1
        float         rPostGain;                    //!< 05Ch  e.g. Gain Stage 2

        // Data from status information ?
        std::int32_t  iLatitude;                    //!< 060h  must be used, UNIT = milli seconds
        std::int32_t  iLongitude;                   //!< 064h  must be used, UNIT = milli seconds
        std::int32_t  iElevation;                   //!< 068h  must be used, UNIT = cm
        char          byLatLongType;                //!< 06Ch  'G' default, 'U' user, GPS should be used
        char          byAddCoordType;               //!< 06Dh  'U' = UTM, default empty
        std::int16_t  siRefMeridian;                //!< 06Eh  default empty, should not be used (external)



        //!@todo can we store 64bit time and 64bit samples here ??
        double        dblNorthing;                  //!< 070h  also xcoord should not be used, default 0 (external)
        double        dblEasting;                   //!< 078h  also ycoord should not be used  default 0 (external)
        char          byGPSStat;                    //!< 080h  '-' unknown, 'N' no fix, 'C' full fix
        char          byGPSAccuracy;                //!< 081h  '0' - not used, 1 in case GF4-Fix & Syrlinks was active (system was synced before Syrlinks took over)
        std::int16_t  iUTCOffset;                   //!< 082h  UTC + iUTCOffset = GPS time for example; can be used for other offsets, not used, default 0
        char          achSystemType[12];            //!< 084h  MMS03e GMS05 ADU06 ADU07 ADU08 ADU09 SPAMMKV SPAMMKIII

        // Data from XML-Job specification
        char          achSurveyHeaderName [12];     //!< 090h  UTF-8 storage  free for usage
        char          achMeasType          [4];     //!< 09Ch  free for usage MT CSMT


        double        DCOffsetCorrValue;            //!< 0A0h  DAC offset double
        std::int8_t   DCOffsetCorrOn;               //!< 0A8h  DC offset was switched on (1) or off(0)
        std::int8_t   InputDivOn;                   //!< 0A9h  inputput divider on(1) off(0); e.g when coil was connected = 1
        std::int16_t  bit_indicator;                //!< 0AAh  0 = 32bit int INTEL byte order, 1 = 64bit INTEL byte order, little endian; since atsheader version 81
        char          achSelfTestResult [2];        //!< 0ACh  'NO' or 'OK'
        std::uint16_t numslices;                    //!< 0AEh  number of slices used (1....1024, 1 is the first, that is list.size() )

        //std::int16_t  calentries    // max 128 entries

        // Were the following fields ever used ?
        std::int16_t  siCalFreqs;                   //!< 0B0h  not used  (external)
        std::int16_t  siCalEntryLength;             //!< 0B2h  not used  (external)
        std::int16_t  siCalVersion;                 //!< 0B4h  not used  (external)
        std::int16_t  siCalStartAddress;            //!< 0B6h  not used, never used  (external)


        char          abyLFFilters [8];             //!< 0B8h  is a bitfield

        char          achUTMZone  [12];             //!< 0C0h  not used  (external)  (formerly abyADU06CalFilename) 32U or 01G, 32UMD7403 : alway NNC od NNCCNNNN
        std::uint32_t uiADUCalTimeDate;             //!< 0CCh  not used  (external)
        char          achSensorCalFilename [12];    //!< 0D0h  not used  ("SENSOR.CAL") (external)
        std::uint32_t uiSensorCalTimeDate;          //!< 0DCh  not used  (external)

        float         rPowerlineFreq1;              //!< 0E0h  e.g. empty  (external)
        float         rPowerlineFreq2;              //!< 0E4h  e.g. empty  (external)
        char          abyHFFilters[8];              //!< 0E8h  is a bitfield


        // IF uiSamples == UINT32_MAX you find
        std::uint64_t uiSamples64bit;               //!< 0F0h  amount of samples (each is a 32bit /64bit int) in the file total of all slices; ONLY used in case uiSamples == UINT32_MAX; else 0; REPLACSES the uiSamples; do not add

        //double        OriginalLSBMV;              //!< 0F0h  NOT USED ANYMORE! orig lsb from selftest without gains; used for ADC values
        float         rExtGain;                     //!< 0F8h  for external satellite box
        char          achADBBoardType[4];           //!< 0FCh  LF HF or MF or BB


        ATSComments_80 tscComment;                  //!< 100h

        // size if comments is total 100h + 100h data  + (512byte comments = 200h) = 400h
        // start tsdata = 400h = 1024th byte
        // start of CEA header is 400h + 3FFh * 20h = 83E0h
        //                        1024 + 1023 * 23  = 33760
    };


    #endif // ATSHEADER_80_DEF_H

    
.. _dblLSBMV_programmer:

LSB 
^^^^^^

LSB == least significant bit


**The dblLSBMV multiplied with the int32_t data values results in the measured values in mV** Gains and others are included in th LSB. |br|
To scale the E field use rPosX1 ... rPosZ2 values and **NOT!** the rDipLength and rAngle; these values are obsolete |br|
metronix software assumes for the magnetic field that in case all rPosX1 ... rPosZ2 are zero, that Hx is pointing North, Hy East and Hz down. |br|


.. _atm_header:

ATM header
^^^^^^^^^^^^^

That is the binary marker file

.. code-block:: c++

  struct atmheader {
    qint16 siHeaderLength;
    qint16 siHeaderVers;
    quint32 iSamples;
  };

Followed by booleans (for each sample); true = will be processed, false not.

.. _calibration_file:

Calibration File
------------------
E.g. the calibration file has to sections: "chopper on" and "chopper off". |br|
First column indicateds frequency in Hz. |br|
Second column amplitude in V/(nT * Hz) |br|
Third phase in degree (0-360, not radians). |br|

Be default the chopper should be ON for recordings with equal/less than 512 Hz. |br|
For chopper on the lower frequencies can be extended by the theroretical function.
For chopper off this is not possible.

The amplitude is normalized by f (as you can see from the units). So for lower frequencies the amplitude
of the MFS-06e converges agains 0.2 V/(nT*Hz) (and gives a horizontal line in a plot for the chopper on data).

For usage in your software you may

 * multiply magnitude by f
 * make a complex number from magnitude and phase
 * if you take the mV data from the ats, multiply calibration by 1000 in case
 * interpolate the calibration data onto your nodes of the FFT
 * divide by the calibration
 




.. code-block:: bash

  Calibration measurement with NumetriQ PSM 3750 - 0.1.100000.1.33 
  Metronix GmbH, Kocherstr. 3, 38120 Braunschweig
                                                
  Magnetometer: MFS06e#727    Date: 17/01/12    Time: 12:19:57
                                          
                                          
  FREQUENCY    MAGNITUDE    PHASE         
  Hz           V/(nT*Hz)    deg           
  Chopper On
  +1.0000E-01  +1.9996E-01  +8.8589E+01   
  +1.2328E-01  +2.0008E-01  +8.8225E+01   
  +1.5199E-01  +2.0014E-01  +8.7796E+01   
  +1.8738E-01  +1.9987E-01  +8.7329E+01   
  +2.3101E-01  +1.9982E-01  +8.6762E+01   
  +2.8480E-01  +1.9984E-01  +8.5986E+01   
  +3.5111E-01  +1.9961E-01  +8.5047E+01   
  +4.3287E-01  +1.9919E-01  +8.3911E+01   
  +5.3367E-01  +1.9863E-01  +8.2537E+01   
  +6.5793E-01  +1.9767E-01  +8.0848E+01   
  +8.1113E-01  +1.9619E-01  +7.8796E+01   
  +1.0000E+00  +1.9432E-01  +7.6298E+01   
  +1.2329E+00  +1.9151E-01  +7.3290E+01   
  +1.5199E+00  +1.8781E-01  +6.9669E+01   
  +1.8738E+00  +1.8220E-01  +6.5470E+01   
  +2.3101E+00  +1.7438E-01  +6.0673E+01   
  +2.8480E+00  +1.6449E-01  +5.5318E+01   
  +3.5112E+00  +1.5217E-01  +4.9534E+01   
  +4.3288E+00  +1.3785E-01  +4.3570E+01   
  +5.3367E+00  +1.2226E-01  +3.7647E+01   
  +6.5793E+00  +1.0622E-01  +3.2028E+01   
  +8.1113E+00  +9.0622E-02  +2.6892E+01   
  +1.0000E+01  +7.6212E-02  +2.2313E+01   
  +1.2329E+01  +6.3445E-02  +1.8389E+01   
  +1.5199E+01  +5.2247E-02  +1.5069E+01   
  +1.8738E+01  +4.2964E-02  +1.2247E+01   
  +2.3101E+01  +3.5147E-02  +9.8945E+00   
  +2.8480E+01  +2.8661E-02  +8.0352E+00   
  +3.5112E+01  +2.3266E-02  +6.4090E+00   
  +4.3288E+01  +1.8976E-02  +5.1069E+00   
  +5.3367E+01  +1.5470E-02  +3.9140E+00   
  +6.5793E+01  +1.2495E-02  +3.0121E+00   
  +8.1113E+01  +1.0154E-02  +2.2737E+00   
  +1.0000E+02  +8.2466E-03  +1.5682E+00   
  +1.2329E+02  +6.6826E-03  +8.7741E-01   
  +1.5199E+02  +5.4120E-03  +3.2423E-01   
  +1.8738E+02  +4.3959E-03  -2.9814E-01   
  +2.3101E+02  +3.5657E-03  -8.7995E-01   
  +2.8480E+02  +2.8912E-03  -1.5233E+00   
  +3.5112E+02  +2.3408E-03  -2.1688E+00   
  +4.3288E+02  +1.9000E-03  -3.0046E+00   
  +5.3367E+02  +1.5399E-03  -3.9069E+00   
  +6.5793E+02  +1.2478E-03  -4.9982E+00   
  +8.1113E+02  +1.0105E-03  -6.2920E+00   
  +1.0000E+03  +8.1795E-04  -7.8551E+00   
  +1.2329E+03  +6.6137E-04  -9.7494E+00   
  +1.5199E+03  +5.3415E-04  -1.2052E+01   
  +1.8738E+03  +4.3092E-04  -1.4858E+01   
  +2.3101E+03  +3.4918E-04  -1.8440E+01   
  +2.8480E+03  +2.7656E-04  -2.3375E+01   
  +3.5112E+03  +2.1821E-04  -2.8876E+01   
  +4.3288E+03  +1.6854E-04  -3.6126E+01   
  +5.3367E+03  +1.2103E-04  -4.4781E+01   
  +6.5793E+03  +8.0732E-05  -4.5047E+01   
  +8.1113E+03  +6.6265E-05  -4.4906E+01   
  +1.0000E+04  +5.3269E-05  -5.2671E+01   
                                          
                                          
  FREQUENCY    MAGNITUDE    PHASE         
  Hz           V/(nT*Hz)    deg           
  Chopper Off
  +1.0000E+00  +1.8929E-01  +1.1098E+02   
  +1.2329E+00  +1.9877E-01  +1.0178E+02   
  +1.5199E+00  +2.0310E-01  +9.2408E+01   
  +1.8738E+00  +2.0205E-01  +8.2979E+01   
  +2.3101E+00  +1.9525E-01  +7.3551E+01   
  +2.8480E+00  +1.8380E-01  +6.4373E+01   
  +3.5112E+00  +1.6822E-01  +5.5546E+01   
  +4.3288E+00  +1.4990E-01  +4.7282E+01   
  +5.3367E+00  +1.3074E-01  +3.9765E+01   
  +6.5793E+00  +1.1170E-01  +3.3138E+01   
  +8.1113E+00  +9.4077E-02  +2.7353E+01   
  +1.0000E+01  +7.8347E-02  +2.2450E+01   
  +1.2329E+01  +6.4595E-02  +1.8346E+01   
  +1.5199E+01  +5.2931E-02  +1.4879E+01   
  +1.8738E+01  +4.3376E-02  +1.2120E+01   
  +2.3101E+01  +3.5403E-02  +9.7871E+00   
  +2.8480E+01  +2.8760E-02  +7.8304E+00   
  +3.5112E+01  +2.3408E-02  +6.1176E+00   
  +4.3288E+01  +1.8972E-02  +4.9934E+00   
  +5.3367E+01  +1.5398E-02  +4.0097E+00   
  +6.5793E+01  +1.2525E-02  +2.9188E+00   
  +8.1113E+01  +1.0166E-02  +2.1264E+00   
  +1.0000E+02  +8.2561E-03  +1.5565E+00   
  +1.2329E+02  +6.6903E-03  +8.5820E-01   
  +1.5199E+02  +5.4239E-03  +2.0043E-01   
  +1.8738E+02  +4.4025E-03  -3.0994E-01   
  +2.3101E+02  +3.5698E-03  -9.1526E-01   
  +2.8480E+02  +2.8946E-03  -1.5153E+00   
  +3.5112E+02  +2.3489E-03  -2.2518E+00   
  +4.3288E+02  +1.9026E-03  -2.9940E+00   
  +5.3367E+02  +1.5419E-03  -3.8998E+00   
  +6.5793E+02  +1.2494E-03  -4.9771E+00   
  +8.1113E+02  +1.0118E-03  -6.2748E+00   
  +1.0000E+03  +8.1898E-04  -7.8198E+00   
  +1.2329E+03  +6.6221E-04  -9.7244E+00   
  +1.5199E+03  +5.3482E-04  -1.2031E+01   
  +1.8738E+03  +4.3142E-04  -1.4824E+01   
  +2.3101E+03  +3.4914E-04  -1.8435E+01   
  +2.8480E+03  +2.7691E-04  -2.3267E+01   
  +3.5112E+03  +2.1863E-04  -2.8766E+01   
  +4.3288E+03  +1.6899E-04  -3.6023E+01   
  +5.3367E+03  +1.2149E-04  -4.4705E+01   
  +6.5793E+03  +8.1011E-05  -4.5075E+01   
  +8.1113E+03  +6.6395E-05  -4.4922E+01   
  +1.0000E+04  +5.3337E-05  -5.2637E+01   

.. _sats_header:

ATSS header
-------------- 

sats stands for advanced timeseries stream. |br|
The stream data itself is stored in *.atss* as *pure* binary data streams, 64 bit double IEEE 754 as most processor architectures use it. |br|
The double is is used by default in Python/NumPy, MatLab, C++ and others.

The JSON header is separated from the stream. This allows a continuous data transmission where the stream data *only* changes at the end. The sync client 
can use "rsync --append" in order to *resume* the data transfer. 

The JSON header can be transferred with the "rsync -z" option (compress) because it is an ASCII file.

In addition a .sql3 (SQLite file version 3) is written into the data directory, containing the log data of the recording.


The filename is splitted by the underscore tag _ with a default length of 3 and a letter bit_indicator


008_C01_R000_TEy_512H.sats ADU-08e data stream recorded with channel 1 as Ey with 512 Hz |br|
067_C00_R001_TEx_8s.sats ADU-08e data stream recorded with channel 0 as Ex with one sample every 4 seconds |br|

.. csv-table::
 :delim: ,

 serial ADU, Channel 0 - N, Run 0 - N, Type (of channel), Sample Frequency or Sample Period, H(ertz) or S(econds), ext
 008, C001, R000, TEy, 512, H, atss
 067, C000, R001, TEx, 8, s, atss
 
.