6.1. Introduction
Observatories send their data to INTERMAGNET via two different routes. For preliminary data products (all types of non-definitive data including quasi-definitive), each observatory is assigned to a Geomagnetic Information Node (GIN). Observatories send their preliminary data to their assigned GIN in ‘near real-time’, i.e. as soon as possible after recording. The process is designed to be automated, so that observatories can upload data to INTERMAGNET as part of their automatic data recording or processing systems. Multiple transmissions of preliminary data are possible, where an observatory is refining the quality of the data shortly after recording. The GIN is responsible for forwarding this data to the INTERMAGNET web site for onward distribution to users. The INTERMAGNET web site is the only place where users have access to data from INTERMAGNET observatories. Observatories send their definitive data to the ftp site of the Paris GIN. This process is expected to be manual and only done once a year (unless there are problems with the data that require resubmission).
6.1.1. Data Formats
A number of data formats are in use in INTERMAGNET. In some cases this diversity exists to support handling of different types of data, in other cases a new format has superseded (or partially superseded) an older one.
6.1.1.1. IMFV1.23 GIN Dissemination Format
Data type supported |
Notes |
|---|---|
Minute mean values |
GINs continue to support this format, but INTERMAGNET no longer distributes data in IMFV1.23. Observatories are encouraged to use the IAGA-2002 format instead. |
This format is fully defined in APPENDIX E3.
6.1.1.2. IMFV2.83 Satellite Transmission Format
Data type supported |
Notes |
|---|---|
Minute mean values |
A format designed to transport compressed data via GOES and METEOSAT satellites. |
This format is fully defined in APPENDIX E1.
6.1.1.3. IAGA-2002
Data type supported |
Notes |
|---|---|
Any regular time-series geomagnetic data |
The preferred format for submission of preliminary data. |
This format is fully defined in APPENDIX E5.
6.1.1.4. INTERMAGNET Archive Format
Data type supported |
Notes |
|---|---|
Definitive and quasi-definitive minute, hourly, daily values and K-indices |
The required format for submission of definitive data. Primarily used in the annual INTERMAGNET Reference Data Set (IRDS). |
This format is fully defined in APPENDIX C1.
6.1.1.5. ImagCDF
Data type supported |
Notes |
|---|---|
High precision 1-second data |
A new (2016) format for submission of 1-second definitive data. |
This format is fully defined in APPENDIX E6.
6.1.1.6. IYF INTERMAGNET Year-Mean File
Data type supported |
Notes |
|---|---|
Annual mean values |
Primarily used in the IRDS. |
This format is fully defined in APPENDIX C3.
6.1.1.7. IBF INTERMAGNET Baseline File
Data type supported |
Notes |
|---|---|
Baseline values and absolute observation data |
Primarily used in the IRDS. |
This format is fully defined in APPENDIX E4.
6.1.2. Data Types
Geomagnetic data is refined over time as the various sources from which it is composed are recorded, combined and verified. Data type in this context describes the state that a set of data values have reached in the process of being published, from raw data which is read directly from one or more sensors to definitive data which is the final product of an observatory to which no further changes are expected. The data formats used with time-series geomagnetic data include (or imply) a data type field in their metadata. This data type field is explained below.
Data type |
Formats where it can be used |
What it means |
|---|---|---|
Reported |
IMFV1.23 (as a metadata field) IMFV2.83 (implied – data in this format can only be ‘Reported’). |
Preliminary data from an observatory that has not had any baseline corrections applied. It may contain spikes and may have missing values. |
Variation |
IAGA-2002 (as a metadata field) |
The data type is not defined in the format definition.It is assumed to contain data to the same definition as the ‘Reported’ data type. |
Adjusted |
IMFV1.23 (as a metadata field) |
Adjusted data may have modifications made to raw data to apply baselines, remove spikes or fill gaps. |
Provisional |
IAGA-2002 (as a metadata field) |
The data type is not defined in the format definition. It is assumed to contain data to the same definition as the ‘Adjusted’ data type. |
Quasi-definitive |
IMFV1.23, IAGA-2002 and IAFV2.11 (as a metadata field) |
Quasi-definitive data are defined as data that have been corrected using provisional baselines. Produced soon after data acquisition, their accuracy is intended to be very close to that of an observatory’s definitive data product. 98% of the differences between quasi- definitive and definitive data monthly mean values should be less than 5nT in (X, Y, Z) orientation. |
Definitive |
IMFV1.23, IAGA-2002 and IAFV2.11 (as a metadata field) and IAF version prior to V2.11 (implied - data in this format can only be ‘Definitive’). |
Observatory data which have been corrected for baseline variations, have had spikes removed and gaps filled where possible. No further change is expected and the quality of the data is such that they would be used for inclusion in observatory year books and for input to the World Data Centers and the IRDS. |
Where software is used to convert between formats (e.g. at GINs):
The Reported data type is assumed to be interchangeable with the Variation data type.
The Adjusted data type is assumed to be interchangeable with the Provisional data type.
The use of data types to describe more than the state of data in the publication process is prone to error. INTERMAGNET is moving to a system of describing the standard that a data set meets and including this description alongside the data to which it applies.
6.1.3. Geomagnetic Components (or Elements)
Geomagnetic data can be described in a number of different orientations. Component codes are used to describe the individual elements of the geomagnetic vector.
X = H*cos(D)
Y = H*sin(D)
Z = F*sin(I)
H = F*cos(I)
H = SQRT(X^2 + Y^2)
F = SQRT(X^2 + Y^2 + Z^2)
tan(D) = Y/X
tan(I) = Z/H
Fig. 6.1 Geomagnetic components
Code |
Description |
|---|---|
X |
North Component. The strength of the magnetic field vector in the geographic north direction (northerly values are positive). |
Y |
East component. The strength of the magnetic field vector in the geographic east direction (easterly values are positive). |
Z |
Vertical intensity. The strength of the magnetic field vector in the vertical direction (Z is positive down and hence negative south of the geomagnetic equator). |
H |
Horizontal intensity. The strength of the magnetic field vector in the horizontal plane along the magnetic meridian. |
D |
Declination or variation. The angle between the magnetic vector and true north (positive east). |
I |
Inclination. The angle between the magnetic vector and the horizontal plane, in degrees of arc (positive below the horizontal). |
F |
Total field intensity. The geomagnetic field strength, calculated from and consistent with XYZ or HDZ field elements. |
6.1.4. Near Real-Time Preliminary Data
INTERMAGNET wishes to make data available to users as soon as possible after it is recorded. Observatories who are members of INTERMAGNET are required to submit their preliminary data within 72 hours of recording. This requirement has been in place since INTERMAGNET was created and was a challenging target in the days before the INTERNET. It is now a minimum requirement. INTERMAGNET wishes to improve its near real-time performance and has set these goals for near real-time performance:
1-second data: Available to users within 30 seconds.
1-minute data: Available to users within 2 minutes.
These are challenging targets and at present (2018) the INTERMAGNET infrastructure is not able to support these targets; the best possible performance is around 15-20 minutes. However observatories are encouraged to submit preliminary data to their assigned GIN as near to real-time as possible. The best way to achieve this is using the web interface to submit data in IAGA-2002 format.
6.1.5. Geomagnetic Information Nodes
INTERMAGNET has a two stage approach to collection and dissemination of non-definitive data. Observatories send their data to one of 5 Geomagnetic Information Nodes (GINs). The GINs then forward data to the INTERMAGNET web site for distribution to users, where it is made available via a web service, a data download application on the web and an FTP site. GINs may make provision for observatories to access their own data once it has been sent to the GIN, but there is no public access to data at the GINs – all public access to data is via the INTERMAGNET web site. Both GINs and the web site keep a permanent copy of all data sent to them – no data is deleted (though it may be overwritten if an observatory sends an update).
INTERMAGNET will not edit any data that an observatory has sent, though it may contact an observatory if problems are detected and may also remove spikes for the purposes of plotting the data. GINs will send monthly reports of the ‘completeness’ of the data received from observatories. The INTERMAGNET web site will send monthly reports to observatories on the requests users have made for their data. The manager at each GIN acts as a point of contact for IMOs to resolve any data transmission and formatting problems.
6.1.5.1. GIN Manager Addresses
Any enquiries to individual GINs should be made to the INTERMAGNET GIN Manager at the following addresses:
6.1.5.2. GIN Email Addresses
Ottawa: charles.blais@canada.ca
Paris: par_gin@ipgp.fr
Golden: aclaycomb@usgs.gov
Edinburgh:e_gin@mail.nmh.ac.uk
6.1.5.3. INTERMAGNET GIN Types
The five INTERMAGNET GINs can be classified into two types, depending on the observatories they handle and the services they offer.
Type |
Description |
GIN |
|---|---|---|
1 |
|
Golden, Ottawa |
2 |
|
Edinburgh, Kyoto, Paris |
New observatories not run by USGS or GSC would normally be assigned to one of the Type 2 GINs.