Abstract:Much biodiversity data is collected worldwide, but it remains challenging to assemble the scattered knowledge for assessing biodiversity status and trends. The concept of Essential Biodiversity Variables (EBVs) was introduced to structure biodiversity monitoring globally, and to harmonize and standardize biodiversity data from disparate sources to capture a minimum set of critical variables required to study, report and manage biodiversity change. Here, we assess the challenges of a 'Big Data' approach to building global EBV data products across taxa and spatiotemporal scales, focusing on species distribution and abundance. The majority of currently available data on species distributions derives from incidentally reported observations or from surveys where presence-only or presence-absence data are sampled repeatedly with standardized protocols. Most abundance data come from opportunistic population counts or from population time series using standardized protocols (e.g. repeated surveys of the same population from single or multiple sites). Enormous complexity exists in integrating these heterogeneous, multi-source data sets across space, time, taxa and different sampling methods. Integration of such data into global EBV data products requires correcting biases introduced by imperfect detection and varying sampling effort, dealing with different spatial resolution and extents, harmonizing measurement units from different data sources or sampling methods, applying statistical tools and models for spatial inter- or extrapolation, and quantifying sources of uncertainty and errors in data and models. To support the development of EBVs by the Group on Earth Observations Biodiversity Observation Network (GEO BON), we identify 11 key workflow steps that will operationalize the process of building EBV data products within and across research infrastructures worldwide. These workflow steps take multiple sequential activities into account, including identification and aggregation of various raw data sources, data quality control, taxonomic name matching and statistical modelling of integrated data. We illustrate these steps with concrete examples from existing citizen science and professional monitoring projects, including eBird, the Tropical Ecology Assessment and Monitoring network, the Living Planet Index and the Baltic Sea zooplankton monitoring. The identified workflow steps are applicable to both terrestrial and aquatic systems and a broad range of spatial, temporal and taxonomic scales. They depend on clear, findable and accessible metadata, and we provide an overview of current data and metadata standards. Several challenges remain to be solved for building global EBV data products: (i) developing tools and models for combining heterogeneous, multi-source data sets and filling data gaps in geographic, temporal and taxonomic coverage, (ii) integrating emerging methods and technologies for data collection such as citizen science, sensor networks, DNA-based techniques and satellite remote sensing, (iii) solving major technical issues related to data product structure, data storage, execution of workflows and the production process/cycle as well as approaching technical interoperability among research infrastructures, (iv) allowing semantic interoperability by developing and adopting standards and tools for capturing consistent data and metadata, and (v) ensuring legal interoperability by endorsing open data or data that are free from restrictions on use, modification and sharing. Addressing these challenges is critical for biodiversity research and for assessing progress towards conservation policy targets and sustainable development goals.

BDcleaner: A workflow for cleaning taxonomic and geographic errors in occurrence data archived in biodiversity databases

A globally synthesised and flagged bee occurrence dataset and cleaning workflow

Geographic And Taxonomic Occurrence R‐based Scrubbing (gatoRs): An R package and workflow for processing biodiversity data

Improper data practices erode the quality of global ecological databases and impede the progress of ecological research

Cleaning by clustering: methodology for addressing data quality issues in biomedical metadata

A new R package to parse plant species occurrence records into unique collection events efficiently reduces data redundancy

An automated data cleaning method for Electronic Health Records by incorporating clinical knowledge

Data Cleaning and Semantic Improvement in Biological Databases

NCycDB: a curated integrative database for fast and accurate metagenomic profiling of nitrogen cycling genes.

Location, location, location: utilizing pipelines and services to more effectively georeference the world's biodiversity data

The retrospective double-entry of a long-term ecological dataset

tombRaider - improved species and haplotype recovery from metabarcoding data through artefact and pseudogene exclusion.

An Ontology-Based Approach to Data Cleaning

BAGS: An automated Barcode, Audit & Grade System for DNA barcode reference libraries

Harmonising digitised herbarium data to enhance biodiversity knowledge: creating an updated checklist for the flora of Greenland.

lab2clean: a novel algorithm for automated cleaning of retrospective clinical laboratory results data for secondary uses

A Systematic Approach to Cleaning Routine Health Surveillance Datasets: An Illustration Using National Vector Borne Disease Control Programme Data of Punjab, India

Out-of-date datasets hamper conservation of species close to extinction

Avibase – a database system for managing and organizing taxonomic concepts

Repositories for Taxonomic Data: Where We Are and What is Missing

Building essential biodiversity variables (EBVs) of species distribution and abundance at a global scale