Field Data Management System

This is a brief overview of the TERN and QLD DES field data management system.

The details and evolution of this system are well documented in Chapter 3 of the TERN Effective Field Calibration and Validation Practices Handbook, so only a brief summary of the system is provided here. The field data management system consists of documentation, software/hardware and processes designed to facilitate the consistent collection, recording, storage and delivery of field data. 

Open

Planning

In the planning stage of data management, data collection protocols are developed, which outline the exact collection details for each data set. Updates to the system's tools (including data collection forms and processing scripts), software/hardware and processes is undertaken. Documentation is a key component of the planning stage. 

Collection/Recording

Field data was collected following standardised protocols developed prior to field data collection in the planning stage.  Supporting ancillary data is collected using standardised field data collection forms.  All data is backed up and physical data, such as hard copy field forms, are stored in central location.

 Collation and Storage

On return from the field, data is collated and any ancillary data recorded on hardcopy field sheets is entered into specialised electronic data forms for upload into the ODK aggregator.  All aggregated data and instrument files are organised into a form enabling the data sets to be processed via specialised Python scripts.  These scripts are designed to upload ancillary data onto the JRSRP PostgreSQL spatial database, as well as rename instrument filenames to fit the TERN filenaming convention.

 Delivery

The JRSRP PostGIS database directly links to the JRSRP GeoServer, which in turn links to the JRSRP Field Data Portal.  Delivery is therefore dynamic, with the information on the portal being the most up-to-date version of any given data set.  Renamed instrument files are zipped and delivered by being placed within a directory linking to the TERN THREDDS server.  All data and relevant information are brought together both within the JRSRP Field Data Portal (through the use of pop up windows specific to each dataset) or within the data set metadata records.  Both mechanisms contain links to all relevant data and metadata.

The field data sets typically consist of instrument files or measurements and a supporting shapefile of ancillary data (such as coordinates, date, other observations).  Associated metadata records for each data set contain details assisting users determine the suitability of the data for their purposes, including abstract, licensing, contact details, spatial and temporal scales etc.  Additional information such as data collection protocols and associated reports are also publicly available.

Most of the information is available by direct download from the portal or via links to online sources.  For instrument data sets (imagery, scans, data files etc), users are directed to the TERN THREDDS server or download via http.