Versions Compared

Old Version 10

changes.mady.by.user Leah Gustafson (Unlicensed)

Saved on

compared with

New Version 11

changes.mady.by.user Leah Gustafson (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Queries take data form different areas and are Using a query to access data from different areas may be a more user friendly option than using endpoints. Output is In addition, the output may be usable without further manipulation (depending on the intended use).

Search queries are independent of the tools that are used for sending the query request. For example, the same query body can be used with a request written in cURL, python, postman, R, or another programming language.

Table of Contents

Remember that all data endpoints allow filtering!

The query parameter in the request body allows the data requested from any endpoint to be filtered according to the parameters specified. For example:

  • From the * endpoint, it is possible to retrieve only sites which contain a feature of interest (FOI) = “plant occurrence”

  • From the * endpoint, it is possible to retrieve observations which contain a parameter = “basal area”

Understanding EcoPlots data queries

One of the strengths of EcoPlots is how powerful the query system is. Data are stored as atomic elements (observations) allowing to filter by 0 to many facets.

Before the user starts to define queries, it is important to understand how the underlying system has been built. Values for every search category store a URI (Uniform Resource Identifier), instead of any human-readable value. In EcoPlots case, this URI is actually a resolvable URL identifying a unique concept or instance of a class.

"http://linked.data.gov.au/dataset/ausplots" is the identifier of ausplots in TERN Surveillance Monitoring (a single instance of the Dataset class)

"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"is the identifier of “plant occurrence” (a * within the Concept class that belongs to the Concept Scheme “Feature types”).

This * design offers multiple benefits. Because a URI is a unique and immutable identifier for a specific concept, we can easily modify the vocabulary by changing the label, the description or any other property of the concept without having to update the data.

The only drawback to using URIs is that they are usually not human readable. However, all our URIs are resolvable using the HTTP protocol, meaning that you can paste the URI into your browser and retrieve all the details of the element.

There are several different ways you can discover our vocabularies (the possible values in queries):

  1. Inspecting the TERN vocabularies, then use the vocabulary URI of interest

  2. Using the /discover/{field} endpoint. This endpoint returns a set of “key: value” (consisting of a label and uri) entries with all the possible values for every facet.

  1. Using the Ecoplots Portal to discover data, then set use appropriate settings to create a query with the API code generator. It can then be used in your API request.

The filter query must be sent in the request body (content type JSON) along with any pagination parameters.

The following examples show how to structure a query and some commonly encountered syntax problems.

Query example: this needs to be rewritten as the full query

Code Block
"query": {
    "dataset": [
        "http://linked.data.gov.au/dataset/ausplots"
    ],
    "feature_type": [
        "http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
    ],
    "observed_property": [
        "http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655"
    ]
}

*Each of these examples is separate

Tip

Valid request bodies

Code Block
{
    "page_number": 1, # optional
    "page_size": 50, # optional
    "query": { # mandatory, but query can be empty
        "dataset": [
            "http://linked.data.gov.au/dataset/ausplots"
        ]
    }
}
Code Block
{
    "page_number": 1,
    "page_size": 50,
    "query": {}
}
Code Block
{
    "query": {}
}

Remember that comments within JSON invalidates the syntax. For first invalid example below, there must be a page number is there is a page size.

Warning

Invalid request bodies

Code Block
{
    "page_number": 1,
    "query": {}
}
Code Block
{}
Code Block
{
    "page_number": 1,
    "page_size": 50
}

Available facets in the query object

Probably need a link to the index of possible query bodies.

Notice that all values (even if it is 1 single value) must be enclosed between square brackets.

Tip

Valid query bodies

Code Block
"dataset": [
    "http://linked.data.gov.au/dataset/ausplots"
]
Code Block
"dataset": [
    "http://linked.data.gov.au/dataset/ausplots",
    "http://linked.data.gov.au/dataset/corveg"
]
Warning

Invalid query bodies

Code Block
"dataset": "http://linked.data.gov.au/dataset/ausplots"

Adding an empty list is a valid request but the response will be always empty.

Code Block
"dataset": []

FROM HERE ON NEEDS SOME WEEDING TO WORK OUT WHAT MIGHT BE THE MOST USEFUL WAY TO SHOW SOME OF THESE EXAMPLES WITHOUT REPEATING EARLIER CONTENT

Region

Filter data whose site is located within a specific region.

Example 1 - Data collected in the Bioregion Cape York Peninsula

Code Block
"region": [
  "http://linked.data.gov.au/dataset/bioregion/CYP"
]

Example 2 - Data collected in the Queensland state

Code Block
"region": [
      "http://linked.data.gov.au/dataset/asgs2016/stateorterritory/3"
    ]

Example 3 - Data collected either in Tasmania or in Cape York Peninsula

Code Block
"region": [
      "http://linked.data.gov.au/dataset/asgs2016/stateorterritory/6",
      "http://linked.data.gov.au/dataset/bioregion/CYP"
    ]
Dataset

Filter data by dataset.

Example - Data from “Tern Surveillance Monitoring” or “Williams Wet Tropics Vertebrates”

Code Block
"dataset": [
  "http://linked.data.gov.au/dataset/ausplots",
  "http://linked.data.gov.au/dataset/wet-tropics-vertebrate"
]
Site

Retrieve data from 1 or more ecological monitoring sites.

Example - Retrieve all the data from Tern Surveillance Monitoring site nsabbs0001

Code Block
"site_id": [
  "http://linked.data.gov.au/dataset/ausplots/site-nsabbs0001"
]
Site Visit

Retrieve data from 1 or more ecological monitoring sites.

Example - Retrieve all the data from Tern Surveillance Monitoring sitevisit 58580

Code Block
"site_visit_id": [
  "http://linked.data.gov.au/dataset/ausplots/sv-58580"
]
Feature type (feature of interest)

EcoPlots observations are categorised in Feature of Interest (FOI). Please see the FOI vocabularies for more detail.

Example - Retrieve all the data from Landform feature of interest

Code Block
"feature_type": [
  "http://linked.data.gov.au/def/tern-cv/2cf3ed29-440e-4a50-9bbc-5aab30df9fcd"
]
Parameter (observable property)

Each observation in EcoPlots is measuring an observable property (or parameter). Please see the Parameter vocabularies for more detail.

Example - Get all the data from Crown cover parameter.

Code Block
"observed_property": [
  "http://linked.data.gov.au/def/tern-cv/83ac0278-51cd-4d1a-8e05-7c1fc68cf020"
]
Temporal extent (date)

In addition to all the vocabularised facets, you can filter data based on a temporal resolution.

Code Block
"date_from": "2006-01-01",
"date_to": "2021-07-30"
Spatial extent

Need to explain how this works and how to do this.

Statial queries in EcoPlots allows three different formats: GeoJSON, Well-Known Text (WKT) and Bounding Box. The spatial query will do an “intersect” between the given geometry and the geometry of the data.

Code Block
# WKT examples:
"spatial": "POINT (-77.03653 38.897676)"
"spatial": "LINESTRING (-77.03653 38.897676, -77.009051 38.889939)"
"spatial": "POLYGON ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0))"
"spatial": "MULTIPOINT (102.0 2.0, 103.0 2.0)"
"spatial": "MULTILINESTRING ((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0), (100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8))"
"spatial": "MULTIPOLYGON (((102.0 2.0, 103.0 2.0, 103.0 3.0, 102.0 3.0, 102.0 2.0)), ((100.0 0.0, 101.0 0.0, 101.0 1.0, 100.0 1.0, 100.0 0.0), (100.2 0.2, 100.8 0.2, 100.8 0.8, 100.2 0.8, 100.2 0.2)))"
Code Block
# GEOJSON examples:
"spatial": {
    "type" : "Point",
    "coordinates" : [-77.03653, 38.897676]
}

"spatial": {
    "type" : "LineString",
    "coordinates" : [[-77.03653, 38.897676], [-77.009051, 38.889939]]
}
  
"spatial": {
    "type" : "Polygon",
    "coordinates" : [
      [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]
    ]
}
  
"spatial": {
    "type" : "MultiPolygon",
    "coordinates" : [
      [ [[102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0], [102.0, 2.0]] ],
      [ [[100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0]],
        [[100.2, 0.2], [100.8, 0.2], [100.8, 0.8], [100.2, 0.8], [100.2, 0.2]] ]
    ]
}
  
"spatial": {
    "type": "MultiPoint",
    "coordinates": [
        [
            139.213700000,
            -34.358210000
        ],
        [
            150,
            -35
        ]
    ]
}
Code Block
# Bounding Box (BBOX) example
"spatial": [139.2, -34.35, 140.0, -35.00]

Pagination

Probably needs to be incorporated into the broader topic rather than being a stand alone entity. Perhaps put the initial explanation the first relevant section.

Note

Pagination is meant to be used as a data discovery tool (quickly returning small chunks of data).

Completeness cannot be guaranteed due to database design and implemented limits (only the first 10,000 elements of the whole subset can be retrieved).

If you desire to retrieve more than 10,000 elements, please remove pagination to get all elements.

The EcoPlots API allows users to paginate their data request in order to retrieve small chunks of data at a time. This is great to discover the data, to inspect the shape of the response and to adjust the desired query to narrow the result subset.

In order to control the amount of computational resources needed to paginate over millions of elements, a limit of 10,000 has been established. This means that even if the count of the query (i.e., the total amount of elements) is greater than 10,000, you will be only able to paginate over and retrieve the first 10,000.

Once the query has been defined, you can remove pagination to retrieve the full set of elements.

Code Block
{
    "page_number": 1,
    "page_size": 50,
    "query": {
        "dataset": [
            "http://linked.data.gov.au/dataset/ausplots"
        ],
        "feature_type": [
            "http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
        ],
        "observed_property": [
            "http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655"
        ]
    }
}

Adding pagination to your request is as easy as include the “page_number” and “page_size” parameters to the request body.

Response headers are returned with information about the pagination:

Code Block
Pagination-Page: 1
Pagination-Size: 50
Pagination-Limit: 200 # Max number of page available. Notice that it could be smaller than the real number of pages due to the 10,000 limitation.
Pagination-Count: 11827 # Total amount of elements

Request example:

Code Block
curl -X POST "https://ecoplots-test.tern.org.au/api/v1.0/observations?dformat=ndjson" -H "accept: */*" -H "Authorization: apikey-v1 ZjF6eUpHaE1uRDFwNmRIUi5UcHVkQVUMVHFoYn5QcU1MeF4KVydnISwtclR3dHpZaSg7dHVFeSgocVcNLEZHQ0ozeSMiUGk/TzRHUClCU0he" -H "Content-Type: application/json" -d "{\"page_num\":1,\"page_size\":50,\"query\":{\"dataset\":[\"http://linked.data.gov.au/dataset/ausplots\"],\"feature_type\":[\"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611\"],\"observed_property\":[\"http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655\"]}}"