NEW_EcoPlots API search queries

Queries take data form different areas and are more user friendly than using endpoints. Output is usable without further manipulation.

Remember that all data endpoints allow filtering!

The query parameter in the request body allows the data requested from any endpoint to be filtered according to the parameters specified. For example:

From the * endpoint, it is possible to retrieve only sites which contain a feature of interest (FOI) = “plant occurrence”
From the * endpoint, it is possible to retrieve observations which contain a parameter = “basal area”

Understanding EcoPlots data queries

One of the strengths of EcoPlots is how powerful the query system is. Data are stored as atomic elements (observations) allowing to filter by 0 to many facets.

Before the user starts to define queries, it is important to understand how the underlying system has been built. Values for every search category store a URI (Uniform Resource Identifier), instead of any human-readable value. In EcoPlots case, this URI is actually a resolvable URL identifying a unique concept or instance of a class.

"http://linked.data.gov.au/dataset/ausplots" is the identifier of ausplots in TERN Surveillance Monitoring (a single instance of the Dataset class)
"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"is the identifier of “plant occurrence” (a * within the Concept class that belongs to the Concept Scheme “Feature types”).

This * design offers multiple benefits. Because a URI is a unique and immutable identifier for a specific concept, we can easily modify the vocabulary by changing the label, the description or any other property of the concept without having to update the data.

The only drawback to using URIs is that they are usually not human readable. However, all our URIs are resolvable using the HTTP protocol, meaning that you can paste the URI into your browser and retrieve all the details of the element.

There are several different ways you can discover our vocabularies (the possible values in queries):

Inspecting the TERN vocabularies, then use the vocabulary URI of interest
Using the /discover/{field} endpoint. This endpoint returns a set of “key: value” (consisting of a label and uri) entries with all the possible values for every facet.

The filter query must be sent in the request body (content type JSON) along with any pagination parameters.

The following examples show how to structure a query and some commonly encountered syntax problems.

Query example: this needs to be rewritten as the full query

"query": {
    "dataset": [
        "http://linked.data.gov.au/dataset/ausplots"
    ],
    "feature_type": [
        "http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
    ],
    "observed_property": [
        "http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655"
    ]
}

*Each of these examples is separate

Valid request bodies

{
    "page_number": 1, # optional
    "page_size": 50, # optional
    "query": { # mandatory, but query can be empty
        "dataset": [
            "http://linked.data.gov.au/dataset/ausplots"
        ]
    }
}

{
    "page_number": 1,
    "page_size": 50,
    "query": {}
}

{
    "query": {}
}

Remember that comments within JSON invalidates the syntax. For first invalid example below, there must be a page number is there is a page size.

Invalid request bodies

{
    "page_number": 1,
    "query": {}
}

{}

{
    "page_number": 1,
    "page_size": 50
}

Available facets in the query object

Probably need a link to the index of possible query bodies.

Notice that all values (even if it is 1 single value) must be enclosed between square brackets.

Valid query bodies

"dataset": [
    "http://linked.data.gov.au/dataset/ausplots"
]

"dataset": [
    "http://linked.data.gov.au/dataset/ausplots",
    "http://linked.data.gov.au/dataset/corveg"
]

Invalid query bodies

"dataset": "http://linked.data.gov.au/dataset/ausplots"

Adding an empty list is a valid request but the response will be always empty.

"dataset": []

FROM HERE ON NEEDS SOME WEEDING TO WORK OUT WHAT MIGHT BE THE MOST USEFUL WAY TO SHOW SOME OF THESE EXAMPLES WITHOUT REPEATING EARLIER CONTENT

Region

Filter data whose site is located within a specific region.

Example 1 - Data collected in the Bioregion Cape York Peninsula

"region": [
  "http://linked.data.gov.au/dataset/bioregion/CYP"
]

Example 2 - Data collected in the Queensland state

"region": [
      "http://linked.data.gov.au/dataset/asgs2016/stateorterritory/3"
    ]

Example 3 - Data collected either in Tasmania or in Cape York Peninsula

"region": [
      "http://linked.data.gov.au/dataset/asgs2016/stateorterritory/6",
      "http://linked.data.gov.au/dataset/bioregion/CYP"
    ]

Dataset

Filter data by dataset.

Example - Data from “Tern Surveillance Monitoring” or “Williams Wet Tropics Vertebrates”

"dataset": [
  "http://linked.data.gov.au/dataset/ausplots",
  "http://linked.data.gov.au/dataset/wet-tropics-vertebrate"
]

Site

Retrieve data from 1 or more ecological monitoring sites.

Example - Retrieve all the data from Tern Surveillance Monitoring site nsabbs0001

"site_id": [
  "http://linked.data.gov.au/dataset/ausplots/site-nsabbs0001"
]

Site Visit

Retrieve data from 1 or more ecological monitoring sites.

Example - Retrieve all the data from Tern Surveillance Monitoring sitevisit 58580

"site_visit_id": [
  "http://linked.data.gov.au/dataset/ausplots/sv-58580"
]

Feature type (feature of interest)

EcoPlots observations are categorised in Feature of Interest (FOI). Please see the FOI vocabularies for more detail.

Example - Retrieve all the data from Landform feature of interest

"feature_type": [
  "http://linked.data.gov.au/def/tern-cv/2cf3ed29-440e-4a50-9bbc-5aab30df9fcd"
]

Parameter (observable property)

Each observation in EcoPlots is measuring an observable property (or parameter). Please see the Parameter vocabularies for more detail.

Example - Get all the data from Crown cover parameter.

"observed_property": [
  "http://linked.data.gov.au/def/tern-cv/83ac0278-51cd-4d1a-8e05-7c1fc68cf020"
]

Temporal extent (date)

In addition to all the vocabularised facets, you can filter data based on a temporal resolution.

"date_from": "2006-01-01",
"date_to": "2021-07-30"

Spatial extent

Need to explain how this works and how to do this.

Statial queries in EcoPlots allows three different formats: GeoJSON, Well-Known Text (WKT) and Bounding Box. The spatial query will do an “intersect” between the given geometry and the geometry of the data.

# WKT examples:
"spatial": "POINT (-77.03653 38.897676)"
"spatial": "LINESTRING (-77.03653 38.897676, -77.009051 38.889939)"
"spatial": "POLYGON ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0))"
"spatial": "MULTIPOINT (102.0 2.0, 103.0 2.0)"
"spatial": "MULTILINESTRING ((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0), (100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8))"
"spatial": "MULTIPOLYGON (((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0, 102.0 2.0)), ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8, 100.2 0.2)))"

# GEOJSON examples:
"spatial": {
    "type" : "Point",
    "coordinates" : [-77.03653, 38.897676]
}

"spatial": {
    "type" : "LineString",
    "coordinates" : [[-77.03653, 38.897676], [-77.009051, 38.889939]]
}
  
"spatial": {
    "type" : "Polygon",
    "coordinates" : [
      [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]
    ]
}
  
"spatial": {
    "type" : "MultiPolygon",
    "coordinates" : [
      [ [[102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0], [102.0, 2.0]] ],
      [ [[100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0]],
        [[100.2, 0.2], [100.8, 0.2], [100.8, 0.8], [100.2, 0.8], [100.2, 0.2]] ]
    ]
}
  
"spatial": {
    "type": "MultiPoint",
    "coordinates": [
        [
            139.213700000,
            -34.358210000
        ],
        [
            150,
            -35
        ]
    ]
}

# Bounding Box (BBOX) example
"spatial": [139.2, -34.35, 140.0, -35.00]

Probably needs to be incorporated into the broader topic rather than being a stand alone entity. Perhaps put the initial explanation the first relevant section.

Pagination is meant to be used as a data discovery tool (quickly returning small chunks of data).

Completeness cannot be guaranteed due to database design and implemented limits (only the first 10,000 elements of the whole subset can be retrieved).

If you desire to retrieve more than 10,000 elements, please remove pagination to get all elements.

The EcoPlots API allows users to paginate their data request in order to retrieve small chunks of data at a time. This is great to discover the data, to inspect the shape of the response and to adjust the desired query to narrow the result subset.

In order to control the amount of computational resources needed to paginate over millions of elements, a limit of 10,000 has been established. This means that even if the count of the query (i.e., the total amount of elements) is greater than 10,000, you will be only able to paginate over and retrieve the first 10,000.

Once the query has been defined, you can remove pagination to retrieve the full set of elements.

{
    "page_number": 1,
    "page_size": 50,
    "query": {
        "dataset": [
            "http://linked.data.gov.au/dataset/ausplots"
        ],
        "feature_type": [
            "http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
        ],
        "observed_property": [
            "http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655"
        ]
    }
}

Adding pagination to your request is as easy as include the “page_number” and “page_size” parameters to the request body.

Response headers are returned with information about the pagination:

Pagination-Page: 1
Pagination-Size: 50
Pagination-Limit: 200 # Max number of page available. Notice that it could be smaller than the real number of pages due to the 10,000 limitation.
Pagination-Count: 11827 # Total amount of elements

Request example:

curl -X POST "https://ecoplots-test.tern.org.au/api/v1.0/observations?dformat=ndjson" -H "accept: */*" -H "Authorization: apikey-v1 ZjF6eUpHaE1uRDFwNmRIUi5UcHVkQVUMVHFoYn5QcU1MeF4KVydnISwtclR3dHpZaSg7dHVFeSgocVcNLEZHQ0ozeSMiUGk/TzRHUClCU0he" -H "Content-Type: application/json" -d "{\"page_num\":1,\"page_size\":50,\"query\":{\"dataset\":[\"http://linked.data.gov.au/dataset/ausplots\"],\"feature_type\":[\"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611\"],\"observed_property\":[\"http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655\"]}}"