...
POSTMAN queries (collection) → Importable to Postman by anyone.
| View file | ||
|---|---|---|
|
| Info |
|---|
Not nested regions is slighly faster in most of the executions |
| View file | ||
|---|---|---|
|
Denormalise attributes
The aim of this recommendation is to get rid of nested fields (which create “hiden” Lucene documents) in order to prevent the uncontrolled growth in the size of the index.
...
| Info |
|---|
Old index is ~0.39 times greater than new index in terms of number of documents |
Data before denormalisation:
| Code Block |
|---|
{
"_index": "plotdata_ecoplots-data-observations-ausplots2-aggregate-plant-20211026050100",
"_type": "_doc",
"_id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348-134833",
"_score": 1,
"_ignored": [
"foi_attributes.value.value.float",
"foi_attributes.value.value.integer",
"foi_attributes.value.value.date"
],
"_source": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348",
...
"foi_attributes": [
{
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-transect-54494348",
"attribute": {
"label": "transect direction",
"value": "http://linked.data.gov.au/def/tern-cv/d82a71d7-677a-45fe-968f-6dad4b8e8488"
},
"value": {
"label": null,
"value": "S1-N1",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
{
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-point_number-54494348",
"attribute": {
"label": "transect point",
"value": "http://linked.data.gov.au/def/tern-cv/08035bb8-d972-4110-b3e9-1939bc972ace"
},
"value": {
"label": null,
"value": "74",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
{
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-species_name-54494348",
"attribute": {
"label": "scientific name",
"value": "http://linked.data.gov.au/def/tern-cv/082c84fe-d923-4162-9b73-294b7a8a2dda"
},
"value": {
"label": null,
"value": "Acacia sibirica",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
}
],
...
} |
Date after denormalisation:
| Code Block |
|---|
{
"_index": "plotdata_ecoplots-data_foi",
"_type": "_doc",
"_id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348-134833",
"_score": 1,
"_source": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-obs-height-54494348",
...
"foi_attr_tern:d82a71d7-677a-45fe-968f-6dad4b8e8488": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-transect-54494348",
"value": {
"label": null,
"value": "S1-N1",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attr_tern:08035bb8-d972-4110-b3e9-1939bc972ace": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-point_number-54494348",
"value": {
"label": null,
"value": "74",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attr_tern:082c84fe-d923-4162-9b73-294b7a8a2dda": {
"id": "http://linked.data.gov.au/dataset/ausplots/point_intercept-attr-species_name-54494348",
"value": {
"label": null,
"value": "Acacia sibirica",
"type": "string"
},
"unit_of_measure": {
"label": null,
"value": null
}
},
"foi_attributes": [
"http://linked.data.gov.au/def/tern-cv/d82a71d7-677a-45fe-968f-6dad4b8e8488",
"http://linked.data.gov.au/def/tern-cv/08035bb8-d972-4110-b3e9-1939bc972ace",
"http://linked.data.gov.au/def/tern-cv/082c84fe-d923-4162-9b73-294b7a8a2dda"
]
...
}
} |
No nested fields at all
Following above practices (regions and attributes) we have got rid of all nested fields in the mapping (this is, regions, foi_attributes, obs_attributes and instr_attributes).
| Info |
|---|
As it was expected, the number of documents in this new index matches the number of observations. |
Total number of fields in alias
Denormalisation means having 1 field (which has sub fields) per attribute.
In order to control the number of fields (as ES has some limitations to mainly avoid “mapping explosion”, as well as internal limits in Lucene), this simple command count the number of fields in an index.
Using wc from mapping
curl -s -XGET https://es-test.tern.org.au/plotdata_ecoplots-data_no_nested/_mapping?pretty --header 'Authorization: Basic TOKEN' | grep type | wc -l
At 1/11/2021 with 1 full dataset ingested, the total number of fields is 261.
| Note |
|---|
Question: Are limits set by index, aren’t they? |
For this testing, all original indices have been transformed and merged into only 1 index, but the real data would be: 1 index per FOI and dataset.
Using _field_caps API
curl -s -XGET "https://es-test.tern.org.au/plotdata_ecoplots-data_no_nested/_field_caps?fields=*" | jq '.fields|length'
At 1/11/2021 with 1 full dataset ingested, the total number of fields is 290.
Denormalise regions
| Code Block |
|---|
...
"region_types": [
"http://linked.data.gov.au/dataset/local-gov-areas-2011",
"http://linked.data.gov.au/dataset/nrm-2017",
"http://linked.data.gov.au/dataset/bioregion/IBRA7",
"http://linked.data.gov.au/dataset/bioregion",
"http://linked.data.gov.au/dataset/asgs2016/stateorterritory",
"http://linked.data.gov.au/dataset/wwf-terr-ecoregions"
],
"region:local-gov-areas-2011": "http://linked.data.gov.au/dataset/local-gov-areas-2011/32250",
"region:nrm-2017": "http://linked.data.gov.au/dataset/nrm-2017/3080",
"region:bioregion/IBRA7": "http://linked.data.gov.au/dataset/bioregion/GUP",
"region:bioregion": "http://linked.data.gov.au/dataset/bioregion/GUP01",
"region:asgs2016/stateorterritory": "http://linked.data.gov.au/dataset/asgs2016/stateorterritory/3",
"region:wwf-terr-ecoregions": "http://linked.data.gov.au/dataset/wwf-terr-ecoregions/12945",
... |
ES document mapping:
| Code Block |
|---|
"region:asgs2016/stateorterritory" : {
"type" : "keyword"
},
"region:bioregion" : {
"type" : "keyword"
},
"region:bioregion/IBRA7" : {
"type" : "keyword"
},
"region:capad-2018-terrestrial" : {
"type" : "keyword"
},
"region:local-gov-areas-2011" : {
"type" : "keyword"
},
"region:nrm-2017" : {
"type" : "keyword"
},
"region:wwf-terr-ecoregions" : {
"type" : "keyword"
},
"region_types" : {
"type" : "keyword"
}, |
Mapping generated dynamically using https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html
| Code Block |
|---|
"mappings" : {
"dynamic_templates" : [
{
"region_as_keyword" : {
"match" : "region:*",
"mapping" : {
"type" : "keyword"
}
}
},
...
]
} |
Force Merge API
Indices segments are merged after every indexing.
Disable refresh during indexing
Disabling index refresh makes indexing times notably faster (thoughput: ~1000 every two seconds).
1 refresh action is performed manually after indexing. Then the index segments are merged (force-merge).
Dynamic mapping
In order to ensure that the correct datatype is stored in ES for each attribute value, dynamic templating is performed during indexing following the defined rules:
| Code Block |
|---|
"mappings" : {
"dynamic_templates" : [
{
"region_as_keyword" : {
"match" : "region:*",
"mapping" : {
"type" : "keyword"
}
}
},
{
"attribute_field" : {
"path_match" : "*_attr_*.attribute",
"mapping" : {
"type" : "keyword"
}
}
},
{
"id_field" : {
"path_match" : "*_attr_*.id",
"mapping" : {
"type" : "keyword"
}
}
},
{
"unit_field" : {
"path_match" : "*_attr_*.unit_of_measure",
"mapping" : {
"type" : "keyword"
}
}
},
{
"value_label_field" : {
"path_match" : "*_attr_*.value.label",
"mapping" : {
"type" : "text"
}
}
},
{
"value_type_field" : {
"path_match" : "*_attr_*.value.type",
"mapping" : {
"type" : "keyword"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_float",
"mapping" : {
"coerce" : true,
"doc_values" : true,
"ignore_malformed" : true,
"type" : "float"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_int",
"mapping" : {
"coerce" : true,
"doc_values" : true,
"ignore_malformed" : true,
"type" : "integer"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_bool",
"mapping" : {
"normalizer" : "lowercase_normalizer",
"type" : "keyword"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_datetime",
"mapping" : {
"format" : "yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||d/MM/yyyy||epoch_millis",
"ignore_malformed" : "true",
"type" : "date"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_date",
"mapping" : {
"format" : "yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||d/MM/yyyy||epoch_millis",
"ignore_malformed" : "true",
"type" : "date"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_uri",
"mapping" : {
"type" : "keyword"
}
}
},
{
"value_value_field" : {
"path_match" : "*_attr_*.value.value_string",
"mapping" : {
"type" : "keyword"
}
}
}
],
...
} |
Clear cache API | Elasticsearch Guide [7.10] | Elastic
Clear cache after each test query to really test performance improvement!!!
Data indices Mapping
| Expand | ||
|---|---|---|
| ||
|
| Expand | ||
|---|---|---|
| ||
|