The EcoPlots API allows users to paginate their data request so they can quickly retrieve small chunks of data. This is great way to discover what data is available, to inspect the shape of the response, and to adjust the desired query to narrow the result subset.
It also acts to manage the computational resources that are needed to paginate many millions of elements for multiple requests. For this reason, a limit of 10,000 elements per request has been established. This means that even if the count of the query (i.e., the total amount of elements) is greater than 10,000, you will be only able to paginate over and retrieve the first 10,000 results.
Once the query has been sufficiently refined, set the "page_size" to -1. This will retrieve all the data, possibly be millions of records, for that query. Long wait times for a response can be expected depending on the size of the records and quality of connection (often 10mins or more).
Using pagination
Pagination is most useful when implemented as part of a data discovery tool. It allows a way of quickly returning small chunks of data so that searches can be refined quickly and efficiently.
Example query body with pagination:
{
"page_number": 1,
"page_size": 20,
"query": {
"feature_type": [
"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
]
}
}
The above example has a page_number of 1 and a page_size of 20, which means that one page of results will be returned that contains twenty elements.
Response headers are returned with information about the pagination:
Pagination-Page: 1 Pagination-Size: 50 Pagination-Limit: 200 # Max number of page available. Notice that it could be smaller than the real number of pages due to the 10,000 limitation. Pagination-Count: 11827 # Total amount of elements
cURL request example:
curl -X POST "https://ecoplots.tern.org.au/api/v1.0/foi/attributes?dformat=ndjson" -H "accept: */*" -H "X-Api-Key: aAbBcC...xXyYzZ" -H "Content-Type: application/json" -d "{\"page_num\":1,\"page_size\":5,\"query\":{\"feature_type\":[\"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611\"]}}""
Once the query has been defined appropriately, remove pagination to retrieve the full set of elements.
Add Comment