Programmers Notes
CSV tables
In some modules CSV tables may by imported (mk_master_cal).
The implemented style is:
serial |
chopper |
f |
a |
p |
stddev a |
stddev p |
1 |
1 |
1.232800e-01 |
2.000200e-01 |
8.823600e+01 |
2.734738e-04 |
2.734738e-04 |
1 |
0 |
1.000000e+00 |
1.892500e-01 |
1.112900e+02 |
1.312173e-03 |
1.312173e-03 |
serial, chopper, f, a, p, stddev a, stddev p
1, 1, 1.232800e-01, 2.000200e-01, 8.823600e+01, 2.734738e-04, 2.734738e-04
1, 0, 1.000000e+00, 1.892500e-01, 1.112900e+02, 1.312173e-03, 1.312173e-03
CVS tables are always TEXT. When performing actions based on tables, be aware that 1.0e+00 and 1.0E+00 are NOT THE SAME
Also in case of mathematics a string must be converted into a number and back:
.. 1.0e+00 -> double is ok, 1.0e+00 -> int is not ok; so internally we always go 1.0e+00 -> double -> int (in case).
In case of string data we always compare lower case!
All frequencies you use have to be sorted from low to high.
ATS header
ats stands for advanced timeseries.
In languages such as C++ you do simlilar like (here with C++ & Qt)
ATSHeader_80 bin_atsheader; //!< binaray header for read/write
....
file.setFileName("myatsfile.ats");
file.open(QIODevice::ReadOnly))
QDataStream qds;
qds.setDevice(file);
qds.readRawData((char*) &bin_atsheader, sizeof(bin_atsheader));
qDebug() << "samples: " << bin_atsheader.uiSamples;
In other languages reading a binary file is maybe similar.
#ifndef ATSHEADER_80_DEF_H
#define ATSHEADER_80_DEF_H
#include <cstdint>
//
//
//
// array of chars are not NULL terminated !
// achChanType [2] = "Ex" is not NULL terminated
//
//
#define C_ATS_CEA_NUM_HEADERS 1023 /*!< maximum number of 1023 headers of the ATS sliced file.*/
/*!
\brief The ATSSliceHeader_s struct
A sliced ats file ALWAYS contains a default header and C_ATS_CEA_NUM_HEADERS (1023) = 1024 total.<br>
The FIRST and DEFAULT slice contains ALL information (total samples of all slices)<br>
The 0 ... 1022 slices (= ATS_CEA_NUM_HEADERS ) contain the individual descriptions<br>
start tsdata = 400h = 1024th byte<br>
start of CEA data is 400h + 3FFh * 20h = 83E0h<br>
1024 + 1023 * 32 = 33760<br>
assuming 3 slices of 4096 iSamples will be 12,288 samples in the header,
followed by 3 slice headers with 4096 iSamples each
and 1019 empty slice headers<br>
the data has to sorted out on/after 83E0h according to the slice info
*/
struct ATSSliceHeader_1080
{
std::uint32_t uiSamples; //!< 000h number of samples for this slice
std::uint32_t uiStartDateTime; //!< 004h startdate/time as UNIX timestamp (UTC)
double dbDCOffsetCorrValue; //!< 008h DC offset correction value in mV
float rPreGain; //!< 010h originally used pre-gain (GainStage1) - LSB is the same for all slices
float rPostGain; //!< 014h originally used post-gain (GainStage2) - LSB is the same for all slices
std::int8_t DCOffsetCorrOn; //!< 018h DC offset correction was on/off for this slice
std::int8_t reserved[7]; //!< 020h reserved bytes to get to word / dword boundary
};
/*! @todo TX info
maybe UTM Zone number
maybe UTM letter
Tx times six double at least
Tx base freq
*/
/*!
\brief The ATSComments_80 struct for ats header 80,81 and 90 <br>
Some software will automatically convert to higher version<br>
Some fields are external - the user has to set then; examples:<br>
northing, easting, UTM Zone : these can be used for high density grids<br>
*/
struct ATSComments_80 {
char achClient [16]; //!< 000h UTF-8 storage, used in EDI
char achContractor [16]; //!< 010h UTF-8 storage, used in EDI
char achArea [16]; //!< 020h UTF-8 storage, used in EDI
char achSurveyID [16]; //!< 030h UTF-8 storage, used in EDI
char achOperator [16]; //!< 040h UTF-8 storage, used in EDI
char achSiteName [112]; //!< 050h UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
char achXmlHeader [64]; //!< 0C0h UTF-8 storage points to the corresponding XML file, containing addtional information for this header; no PATH=XML file is in the same directory as the ats file
// deleted September 2018 char achComments [512]; //!< 100h UTF-8 storage comment block starts (comment field can start with "weather:" but has no meaning
// new September 2018
char achComments [288]; //!< 100h UTF-8 storage comment block starts (comment field can start with "weather:" but has no meaning
char achSiteNameRR [112]; //!< 220h UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
char achSiteNameEMAP [112]; //!< 290h UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars); INDICATES a default EMAP center station, where we expect the H (magnetic)
};
/*!
\brief The ATSHeader_80 struct<br>
reference is lat and long; additional entries like northing and easting should be empty // not used<br>
these entries can be overwritten by the user in case of high density grids <br>
The site_name (site name, achSiteName, is in the comment field; the site_no, site_num, site_number, site number is nver stored; this is part of external numeration
*/
struct ATSHeader_80 {
std::uint16_t uiHeaderLength; //!< 000h length of header in bytes (default 1024 and 33760 for CEA)
std::int16_t siHeaderVers; //!< 002h 80 for ats, 81 for 64bit possible / metronix, 1080 for CEA / sliced header
// This information can be found in the ChannelTS datastructure
std::uint32_t uiSamples; //!< 004h amount of samples (each is a 32bit / 64bit int INTEL byte order little endian) in the file total of all slices; if uiSamples == UINT32_MAX, uiSamples64bit at address 0F0h will be used instead; uiSamples64bit replaces uiSamples in that case; do not add!
//!<
float rSampleFreq; //!< 008h sampling frequency in Hz
std::uint32_t uiStartDateTime; //!< 00Ch unix TIMESTAMP (some computers will revert that to negative numbers if this number is grater than 32bit signed); 2106-02-07T06:28:14 is the END of this format
double dblLSBMV; //!< 010h least significant bit in mV ()
std::int32_t iGMTOffset; //!< 018h not used, default 0; can be used as "UTC to GMT"
float rOrigSampleFreq; //!< 01Ch sampling frequency in Hz as ORIGINALLY recorded; this value should NOT change (for example after filtering)
//The required data could probably found in the HardwareConfig
std::uint16_t uiADUSerNum; //!< 020h Serial number of the system (logger)
std::uint16_t uiADCSerNum; //!< 022h Serial number of the ADC board (ADB)
std::uint8_t uiChanNo; //!< 024h Channel number
std::uint8_t uiChopper; //!< 025h Chopper On (1) / Off (0); e.g. chopper is on for 512Hz and lower
// Data from XML Job-specification
char achChanType [2]; //!< 026h (Ex, Ey, Ez, Hx, Hy, Hz, Jx, Jy, Jz, Px, Py, Pz, Rx, Ry, Rz and so on)
char achSensorType [6]; //!< 028h (MFS06 MFS06e MFS07 MFS07e MFS10e SHFT02 SHF02e FGS02 FGS03 FGS03e etc. e.g. the "-" in MFS-06e is skipped)
std::int16_t siSensorSerNum; //!< 02Eh Serial number of connected sensor
float rPosX1; //!< 030h e.g. South negative [m]
float rPosY1; //!< 034h e.g. West negative [m]
float rPosZ1; //!< 038h e.g. top, sky [m]
float rPosX2; //!< 03Ch e.g. North positive [m]
float rPosY2; //!< 040h e.g. East positive [m]
float rPosZ2; //!< 044h e.g. bottom, earth [m]
// not used any more use pos values!!; GUI interfaces using Length and direction MUST calculate pos x,y,z and set above.
float rDipLength; //!< 048h e.g. to be calculated; should not be used - my be over written in FUTURE
// not used any more use pos values!!
float rAngle; //!< 04Ch e.g. to be calculated; should not be used - my be over written in FUTURE
// Data from Selftest
float rProbeRes; //!< 050h [ohm]
float rDCOffset; //!< 054h [mV]
float rPreGain; //!< 058h e.g. Gain Stage 1
float rPostGain; //!< 05Ch e.g. Gain Stage 2
// Data from status information ?
std::int32_t iLatitude; //!< 060h must be used, UNIT = milli seconds
std::int32_t iLongitude; //!< 064h must be used, UNIT = milli seconds
std::int32_t iElevation; //!< 068h must be used, UNIT = cm
char byLatLongType; //!< 06Ch 'G' default, 'U' user, GPS should be used
char byAddCoordType; //!< 06Dh 'U' = UTM, default empty
std::int16_t siRefMeridian; //!< 06Eh default empty, should not be used (external)
//!@todo can we store 64bit time and 64bit samples here ??
double dblNorthing; //!< 070h also xcoord should not be used, default 0 (external)
double dblEasting; //!< 078h also ycoord should not be used default 0 (external)
char byGPSStat; //!< 080h '-' unknown, 'N' no fix, 'C' full fix
char byGPSAccuracy; //!< 081h '0' - not used, 1 in case GF4-Fix & Syrlinks was active (system was synced before Syrlinks took over)
std::int16_t iUTCOffset; //!< 082h UTC + iUTCOffset = GPS time for example; can be used for other offsets, not used, default 0
char achSystemType[12]; //!< 084h MMS03e GMS05 ADU06 ADU07 ADU08 ADU09 SPAMMKV SPAMMKIII
// Data from XML-Job specification
char achSurveyHeaderName [12]; //!< 090h UTF-8 storage free for usage
char achMeasType [4]; //!< 09Ch free for usage MT CSMT
double DCOffsetCorrValue; //!< 0A0h DAC offset double
std::int8_t DCOffsetCorrOn; //!< 0A8h DC offset was switched on (1) or off(0)
std::int8_t InputDivOn; //!< 0A9h inputput divider on(1) off(0); e.g when coil was connected = 1
std::int16_t bit_indicator; //!< 0AAh 0 = 32bit int INTEL byte order, 1 = 64bit INTEL byte order, little endian; since atsheader version 81
char achSelfTestResult [2]; //!< 0ACh 'NO' or 'OK'
std::uint16_t numslices; //!< 0AEh number of slices used (1....1024, 1 is the first, that is list.size() )
//std::int16_t calentries // max 128 entries
// Were the following fields ever used ?
std::int16_t siCalFreqs; //!< 0B0h not used (external)
std::int16_t siCalEntryLength; //!< 0B2h not used (external)
std::int16_t siCalVersion; //!< 0B4h not used (external)
std::int16_t siCalStartAddress; //!< 0B6h not used, never used (external)
char abyLFFilters [8]; //!< 0B8h is a bitfield
char achUTMZone [12]; //!< 0C0h not used (external) (formerly abyADU06CalFilename) 32U or 01G, 32UMD7403 : alway NNC od NNCCNNNN
std::uint32_t uiADUCalTimeDate; //!< 0CCh not used (external)
char achSensorCalFilename [12]; //!< 0D0h not used ("SENSOR.CAL") (external)
std::uint32_t uiSensorCalTimeDate; //!< 0DCh not used (external)
float rPowerlineFreq1; //!< 0E0h e.g. empty (external)
float rPowerlineFreq2; //!< 0E4h e.g. empty (external)
char abyHFFilters[8]; //!< 0E8h is a bitfield
// IF uiSamples == UINT32_MAX you find
std::uint64_t uiSamples64bit; //!< 0F0h amount of samples (each is a 32bit /64bit int) in the file total of all slices; ONLY used in case uiSamples == UINT32_MAX; else 0; REPLACSES the uiSamples; do not add
//double OriginalLSBMV; //!< 0F0h NOT USED ANYMORE! orig lsb from selftest without gains; used for ADC values
float rExtGain; //!< 0F8h for external satellite box
char achADBBoardType[4]; //!< 0FCh LF HF or MF or BB
ATSComments_80 tscComment; //!< 100h
// size if comments is total 100h + 100h data + (512byte comments = 200h) = 400h
// start tsdata = 400h = 1024th byte
// start of CEA header is 400h + 3FFh * 20h = 83E0h
// 1024 + 1023 * 23 = 33760
};
#endif // ATSHEADER_80_DEF_H
LSB
LSB == least significant bit
The dblLSBMV multiplied with the int32_t data values results in the measured values in mV Gains and others are included in th LSB.
To scale the E field use rPosX1 … rPosZ2 values and NOT! the rDipLength and rAngle; these values are obsolete
metronix software assumes for the magnetic field that in case all rPosX1 … rPosZ2 are zero, that Hx is pointing North, Hy East and Hz down.
ATM header
That is the binary marker file
struct atmheader {
qint16 siHeaderLength;
qint16 siHeaderVers;
quint32 iSamples;
};
Followed by booleans (for each sample); true = will be processed, false not.
Calibration File
E.g. the calibration file has to sections: “chopper on” and “chopper off”.
First column indicateds frequency in Hz.
Second column amplitude in V/(nT * Hz)
Third phase in degree (0-360, not radians).
Be default the chopper should be ON for recordings with equal/less than 512 Hz.
For chopper on the lower frequencies can be extended by the theroretical function.
For chopper off this is not possible.
The amplitude is normalized by f (as you can see from the units). So for lower frequencies the amplitude of the MFS-06e converges agains 0.2 V/(nT*Hz) (and gives a horizontal line in a plot for the chopper on data).
For usage in your software you may
multiply magnitude by f
make a complex number from magnitude and phase
if you take the mV data from the ats, multiply calibration by 1000 in case
interpolate the calibration data onto your nodes of the FFT
divide by the calibration
Calibration measurement with NumetriQ PSM 3750 - 0.1.100000.1.33
Metronix GmbH, Kocherstr. 3, 38120 Braunschweig
Magnetometer: MFS06e#727 Date: 17/01/12 Time: 12:19:57
FREQUENCY MAGNITUDE PHASE
Hz V/(nT*Hz) deg
Chopper On
+1.0000E-01 +1.9996E-01 +8.8589E+01
+1.2328E-01 +2.0008E-01 +8.8225E+01
+1.5199E-01 +2.0014E-01 +8.7796E+01
+1.8738E-01 +1.9987E-01 +8.7329E+01
+2.3101E-01 +1.9982E-01 +8.6762E+01
+2.8480E-01 +1.9984E-01 +8.5986E+01
+3.5111E-01 +1.9961E-01 +8.5047E+01
+4.3287E-01 +1.9919E-01 +8.3911E+01
+5.3367E-01 +1.9863E-01 +8.2537E+01
+6.5793E-01 +1.9767E-01 +8.0848E+01
+8.1113E-01 +1.9619E-01 +7.8796E+01
+1.0000E+00 +1.9432E-01 +7.6298E+01
+1.2329E+00 +1.9151E-01 +7.3290E+01
+1.5199E+00 +1.8781E-01 +6.9669E+01
+1.8738E+00 +1.8220E-01 +6.5470E+01
+2.3101E+00 +1.7438E-01 +6.0673E+01
+2.8480E+00 +1.6449E-01 +5.5318E+01
+3.5112E+00 +1.5217E-01 +4.9534E+01
+4.3288E+00 +1.3785E-01 +4.3570E+01
+5.3367E+00 +1.2226E-01 +3.7647E+01
+6.5793E+00 +1.0622E-01 +3.2028E+01
+8.1113E+00 +9.0622E-02 +2.6892E+01
+1.0000E+01 +7.6212E-02 +2.2313E+01
+1.2329E+01 +6.3445E-02 +1.8389E+01
+1.5199E+01 +5.2247E-02 +1.5069E+01
+1.8738E+01 +4.2964E-02 +1.2247E+01
+2.3101E+01 +3.5147E-02 +9.8945E+00
+2.8480E+01 +2.8661E-02 +8.0352E+00
+3.5112E+01 +2.3266E-02 +6.4090E+00
+4.3288E+01 +1.8976E-02 +5.1069E+00
+5.3367E+01 +1.5470E-02 +3.9140E+00
+6.5793E+01 +1.2495E-02 +3.0121E+00
+8.1113E+01 +1.0154E-02 +2.2737E+00
+1.0000E+02 +8.2466E-03 +1.5682E+00
+1.2329E+02 +6.6826E-03 +8.7741E-01
+1.5199E+02 +5.4120E-03 +3.2423E-01
+1.8738E+02 +4.3959E-03 -2.9814E-01
+2.3101E+02 +3.5657E-03 -8.7995E-01
+2.8480E+02 +2.8912E-03 -1.5233E+00
+3.5112E+02 +2.3408E-03 -2.1688E+00
+4.3288E+02 +1.9000E-03 -3.0046E+00
+5.3367E+02 +1.5399E-03 -3.9069E+00
+6.5793E+02 +1.2478E-03 -4.9982E+00
+8.1113E+02 +1.0105E-03 -6.2920E+00
+1.0000E+03 +8.1795E-04 -7.8551E+00
+1.2329E+03 +6.6137E-04 -9.7494E+00
+1.5199E+03 +5.3415E-04 -1.2052E+01
+1.8738E+03 +4.3092E-04 -1.4858E+01
+2.3101E+03 +3.4918E-04 -1.8440E+01
+2.8480E+03 +2.7656E-04 -2.3375E+01
+3.5112E+03 +2.1821E-04 -2.8876E+01
+4.3288E+03 +1.6854E-04 -3.6126E+01
+5.3367E+03 +1.2103E-04 -4.4781E+01
+6.5793E+03 +8.0732E-05 -4.5047E+01
+8.1113E+03 +6.6265E-05 -4.4906E+01
+1.0000E+04 +5.3269E-05 -5.2671E+01
FREQUENCY MAGNITUDE PHASE
Hz V/(nT*Hz) deg
Chopper Off
+1.0000E+00 +1.8929E-01 +1.1098E+02
+1.2329E+00 +1.9877E-01 +1.0178E+02
+1.5199E+00 +2.0310E-01 +9.2408E+01
+1.8738E+00 +2.0205E-01 +8.2979E+01
+2.3101E+00 +1.9525E-01 +7.3551E+01
+2.8480E+00 +1.8380E-01 +6.4373E+01
+3.5112E+00 +1.6822E-01 +5.5546E+01
+4.3288E+00 +1.4990E-01 +4.7282E+01
+5.3367E+00 +1.3074E-01 +3.9765E+01
+6.5793E+00 +1.1170E-01 +3.3138E+01
+8.1113E+00 +9.4077E-02 +2.7353E+01
+1.0000E+01 +7.8347E-02 +2.2450E+01
+1.2329E+01 +6.4595E-02 +1.8346E+01
+1.5199E+01 +5.2931E-02 +1.4879E+01
+1.8738E+01 +4.3376E-02 +1.2120E+01
+2.3101E+01 +3.5403E-02 +9.7871E+00
+2.8480E+01 +2.8760E-02 +7.8304E+00
+3.5112E+01 +2.3408E-02 +6.1176E+00
+4.3288E+01 +1.8972E-02 +4.9934E+00
+5.3367E+01 +1.5398E-02 +4.0097E+00
+6.5793E+01 +1.2525E-02 +2.9188E+00
+8.1113E+01 +1.0166E-02 +2.1264E+00
+1.0000E+02 +8.2561E-03 +1.5565E+00
+1.2329E+02 +6.6903E-03 +8.5820E-01
+1.5199E+02 +5.4239E-03 +2.0043E-01
+1.8738E+02 +4.4025E-03 -3.0994E-01
+2.3101E+02 +3.5698E-03 -9.1526E-01
+2.8480E+02 +2.8946E-03 -1.5153E+00
+3.5112E+02 +2.3489E-03 -2.2518E+00
+4.3288E+02 +1.9026E-03 -2.9940E+00
+5.3367E+02 +1.5419E-03 -3.8998E+00
+6.5793E+02 +1.2494E-03 -4.9771E+00
+8.1113E+02 +1.0118E-03 -6.2748E+00
+1.0000E+03 +8.1898E-04 -7.8198E+00
+1.2329E+03 +6.6221E-04 -9.7244E+00
+1.5199E+03 +5.3482E-04 -1.2031E+01
+1.8738E+03 +4.3142E-04 -1.4824E+01
+2.3101E+03 +3.4914E-04 -1.8435E+01
+2.8480E+03 +2.7691E-04 -2.3267E+01
+3.5112E+03 +2.1863E-04 -2.8766E+01
+4.3288E+03 +1.6899E-04 -3.6023E+01
+5.3367E+03 +1.2149E-04 -4.4705E+01
+6.5793E+03 +8.1011E-05 -4.5075E+01
+8.1113E+03 +6.6395E-05 -4.4922E+01
+1.0000E+04 +5.3337E-05 -5.2637E+01
ATSS header
sats stands for advanced timeseries stream.
The stream data itself is stored in .atss as pure binary data streams, 64 bit double IEEE 754 as most processor architectures use it.
The double is is used by default in Python/NumPy, MatLab, C++ and others.
The JSON header is separated from the stream. This allows a continuous data transmission where the stream data only changes at the end. The sync client can use “rsync –append” in order to resume the data transfer.
The JSON header can be transferred with the “rsync -z” option (compress) because it is an ASCII file.
In addition a .sql3 (SQLite file version 3) is written into the data directory, containing the log data of the recording.
The filename is splitted by the underscore tag _ with a default length of 3 and a letter bit_indicator
008_C01_R000_TEy_512H.sats ADU-08e data stream recorded with channel 1 as Ey with 512 Hz
067_C00_R001_TEx_8s.sats ADU-08e data stream recorded with channel 0 as Ex with one sample every 4 seconds
serial ADU |
Channel 0 - N |
Run 0 - N |
Type (of channel) |
Sample Frequency or Sample Period |
H(ertz) or S(econds) |
ext |
008 |
C001 |
R000 |
TEy |
512 |
H |
atss |
067 |
C000 |
R001 |
TEx |
8 |
s |
atss |
.