Pagination
Probably needs to be incorporated into the broader topic rather than being a stand alone entity. Perhaps put the initial explanation the first relevant section.
| Note |
|---|
Pagination is meant to be used as a data discovery tool (quickly returning small chunks of data). Completeness cannot be guaranteed due to database design and implemented limits (only the first 10,000 elements of the whole subset can be retrieved). If you desire to retrieve more than 10,000 elements, please remove pagination to get all elements. |
The EcoPlots API allows users to paginate their data request in order to so they can quickly retrieve small chunks of data at a time. This is great way to discover the what data is available, to inspect the shape of the response, and to adjust the desired query to narrow the result subset.
In order to control the amount of It also acts to manage the computational resources that are needed to paginate over many millions of elements for multiple requests. For this reason, a limit of 10,000 elements per request has been established. This means that even if the count of the query (i.e., the total amount of elements) is greater than 10,000, you will be only able to paginate over and retrieve the first 10,000 results.
Once the query has been definedsufficiently refined, and you desire to retrive all the available items, remove "page_size" and "page_number" from the request body. This will retrieve all the data, possibly be millions of records, for that query. Long wait times for a response can be expected depending on the size of the records and quality of connection (often 10mins or more).
...
Using pagination
Pagination is most useful when implemented as part of a data discovery tool. It allows a way of quickly returning small chunks of data so that searches can be refined quickly and efficiently.
Example query body with pagination:
| Code Block |
|---|
{
"page_number": 1,
"page_size": 5,
"query": {
"feature_type": [
"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611"
]
}
} |
The above example has a page_number of 1 and a page_size of 20, which means that one page of results will be returned that contains twenty elements.
Some platforms will also include response headers with information about the pagination settings that have been used:
| Code Block |
|---|
Pagination-Page: 1
Pagination-Size: 50
Pagination-Limit: 200 # Max number of page available. Notice that it could be smaller than the real number of pages due to the 10,000 limitation.
Pagination-Count: 11827 # Total amount of elements |
cURL request example:
| Code Block |
|---|
curl -X POST "https://ecoplots.tern.org.au/api/v1.0/foi/attributes?dformat=ndjson" -H "accept: */*" -H "X-Api-Key: aAbBcC...xXyYzZ" -H "Content-Type: application/json" -d "{\"page_number\":1,\"page_size\":5,\"query\":{\"feature_type\":[\"http://linked.data.gov.au/def/tern-cv/b311c0d3-4a1a-4932-a39c-f5cdc1afa611\"]}}" |
Once the query has been defined appropriately, remove pagination to retrieve the full set of elements.
...