Taxa Table

Source: CORVEG.TAXA

Example row:

Interpretation in SKOS

1. Create a skos:ConceptScheme for taxa skos:Concept in both the CORVEG and the Australian Plant Census (APC).

2. Link between the mapped concepts in CORVEG and the APC by skos:closeMatch.

Background

The Australian Plant Census (APC) is considered to be the authoritative source of data on vascular plants and mosses in Australia, provided by Biodiversity. They deliver the Australian Plant Name Index (APNI) as a dataset to tie in with the APC.

• https://biodiversity.org.au/nsl/services/export/index - APC and APNI for vascular plants

• https://moss.biodiversity.org.au/nsl/services/export/index - APC and APNI for mosses

Outcome

The terms in APC and CORVEG have been modelled using SKOS and the Darwin Core terms, and a relationship has been created between the two skos:ConceptScheme using skos:closeMatch.

The result of the ETL process for Taxa is three new controlled vocabularies:

• the set of terms from the CORVEG Taxa table

• the set of terms name-matched from the CORVEG Taxa table to the APC terms

• a set of taxon rank terms used in the APC and APNI

Further Information

The mappings for APNI and APC terms have been documented in https://www.anbg.gov.au/ibis25/display/NSL/Data+Extracts+-+Darwin+Core by Biodiversity. The mappings have been created in taxa.py for TERN's use since the Biodiversity's Linked Data APIs are currently down (as of March, 2019).

Due to CORVEG’s Taxa table's column infra_rank using the abbreviated form as its value, a controlled vocabulary had to be created to marry the terms up with APNI (as APNI uses the non-abbreviated form for its taxon rank value). For example, CORVEG’s value is var. while APNI's value is Varietas, but they're the same term.

RDF Example of the Taxon Rank Vocabulary

RDF Example

An example of what the output RDF in Turtle looks for a single taxon Gynura drymophila var. drymophila mapped to APC's Gynura drymophila (F.Muell.) F.G.Davies var. drymophila. This skos:closeMatch mapping is done by mapping the Corveg's taxon name to the canonicalName in APC.

Algorithm

• Create the taxon rank controlled vocabulary (as it is a dependent of the taxa controlled vocabulary)

  • Load APNI CSV

  • Gather the unique taxon ranks

  • Clean up some of the string literals (redundant square brackets)

  • Map to SKOS RDF

• Create the taxa controlled vocabulary, forming two skos:ConceptScheme, one as concepts from CORVEG, the other from APC

  • Create file-based indexes using Whoosh library for APC CSV

  • Load the indexes

    • Note: once indexes have been created, new CSVs won't be indexed unless the force_taxa_index attribute in the YAML file for Taxa table is true.

    • The indexes are created so that searching for the taxon terms is extremely fast compared (otherwise it'd be O(n²) using two loops in time complexity)

  • For each taxon term in CORVEG, create it as a skos:Concept, match its infra_rank to the skos:altLabel of the taxon rank controlled vocabulary

    • Map it to terms in SKOS, DCT, and Darwin Core Terms

  • Do a skos:closeMatch to the same term in APC, name-matching on its canonicalName (with slight fuzzy-matching capabilities of the Whoosh library)

  • Ensure that the taxonRank column in APC is also matched to the taxon rank controlled vocabulary, name-matching on the skos:prefLabel

Comments

Many of the columns' values in APC and APNI can/should be turned into a controlled vocabulary. Currently only the taxon rank values in APNI have been turned into a controlled vocabulary (for interoperability between CORVEG taxon rank values and APNI taxon rank values). Creating controlled vocabularies for the other columns should be a good enhancement in the future but for now, the values are just string literals.

Currently 2225 out of 103,108 CORVEG taxon terms are not mapped to external sources (like APC). It may be possible to map these 2225 terms to other Australian government agencies' datasets in the future, but currently having most of the taxon terms mapped to APC is good enough. A possible way to find out what sources have data on these terms can be done by searching the missing terms on Atlas of Living Australia (ALA), and checking where their source information is from. taxa_terms_not_found.txt is a list of taxon terms that were not name-matched to external sources.

The Python library Whoosh was used to create binary (pickle) file-based indexes so that full text search queries were made possible on the large APC and APNI CSV files.

ALA's sources are not updated often enough (some are two years out of date) and they mix and match data between different sources, which is controversial, therefore we are not using ALA as an external source for extra taxonomy information.

Visual Diagram of a Single Taxon

A visual diagram showcasing the above RDF Turtle snippet.

Open

Action Items