...
It also acts to manage the computational resources that are needed to paginate many millions of elements for multiple requests. For this reason, a limit of 10,000 elements per request has been established. This means that even if the count of the query (i.e., the total amount of elements) is greater than 10,000, you will be only able to paginate over and retrieve the first 10,000 results.If you need to retrieve more than 10,000 elements, please remove pagination to get all elements
Once the query has been sufficiently refined, and you desire to retrive all the available items, remove "page_size" and "page_number" from the request body. This will retrieve all the data, possibly be millions of records, for that query. Long wait times for a response can be expected depending on the size of the records and quality of connection (often 10mins or more).
...
Using pagination
Pagination is most useful when implemented as part of a data discovery tool. It allows a way of quickly returning small chunks of data so that searches can be refined quickly and efficiently.
Example query body with pagination:
| Code Block |
|---|
{
"page_number": 1,
"page_size": 505,
"query": {
"dataset": [
"http://linked.data.gov.au/dataset/ausplots"
],
"feature_type": [
"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
],
"observed_property": [
"http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655"
]
}
} |
Adding pagination to your request is as easy as include the “page_number” and “page_size” parameters to the request body.
Response headers are returned The above example has a page_number of 1 and a page_size of 20, which means that one page of results will be returned that contains twenty elements.
Some platforms will also include response headers with information about the pagination settings that have been used:
| Code Block |
|---|
Pagination-Page: 1 Pagination-Size: 50 Pagination-Limit: 200 # Max number of page available. Notice that it could be smaller than the real number of pages due to the 10,000 limitation. Pagination-Count: 11827 # Total amount of elements |
Request cURL request example:
| Code Block |
|---|
curl -X POST "https://ecoplots-test.tern.org.au/api/v1.0/observationsfoi/attributes?dformat=ndjson" -H "accept: */*" -H "Authorization: apikey-v1 ZjF6eUpHaE1uRDFwNmRIUi5UcHVkQVUMVHFoYn5QcU1MeF4KVydnISwtclR3dHpZaSg7dHVFeSgocVcNLEZHQ0ozeSMiUGk/TzRHUClCU0heX-Api-Key: aAbBcC...xXyYzZ" -H "Content-Type: application/json" -d "{\"page_numnumber\":1,\"page_size\":505,\"query\":{\"dataset\":[\"http://linked.data.gov.au/dataset/ausplots\"],\"feature_type\":[\"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611\"],\"observed_property\":[\"http://linked.data.gov.au/def/tern-cv/0bbd7fcd-0782-4efc-96e6-1f0f7669c655\"]}}" |
Once the query has been defined appropriately, remove pagination to retrieve the full set of elements.
...