...
| Code Block |
|---|
{
"_index": "plotdata_ecoplots-data_foi",
"_type": "_doc",
"_id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348-134833",
"_score": 1,
"_source": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348",
...
"foi_attr_tern:d82a71d7-677a-45fe-968f-6dad4b8e8488": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-transect-54494348",
"value": {
"label": null,
"value": "S1-N1",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attr_tern:08035bb8-d972-4110-b3e9-1939bc972ace": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-point_number-54494348",
"value": {
"label": null,
"value": "74",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attr_tern:082c84fe-d923-4162-9b73-294b7a8a2dda": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-species_name-54494348",
"value": {
"label": null,
"value": "Acacia sibirica",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attributes": [
"http://linked.data.gov.au/def/tern-cv/d82a71d7-677a-45fe-968f-6dad4b8e8488",
"http://linked.data.gov.au/def/tern-cv/08035bb8-d972-4110-b3e9-1939bc972ace",
"http://linked.data.gov.au/def/tern-cv/082c84fe-d923-4162-9b73-294b7a8a2dda"
]
...
}
} |
No nested fields at all
Following above practices (regions and attributes) we have got rid of all nested fields in the mapping (this is, regions, foi_attributes, obs_attributes and instr_attributes).
| Info |
|---|
As it was expected, the number of documents in the index is matches the number of observations. |
Total number of fields in alias
Denormalisation means having 1 field (which has sub fields) per attribute.
In order to control the number of fields (as ES has some limitations to mainly avoid “mapping explosion”, as well as internal limits in Lucene), this simple command count the number of fields in an index.
curl -s -XGET https://es-test.tern.org.au/plotdata_ecoplots-data_no_nested/_mapping?pretty --header 'Authorization: Basic TOKEN' | grep type | wc -l
At 1/11/2021 with 1 full dataset ingested, the total number of fields is 261.
| Note |
|---|
Question: Are limits set by index, aren’t they? |
For this testing, all original indices have been transformed and merged into only 1 index, but the real data would be: 1 index per FOI and dataset.