Abstract:Context. Astronomical data reduction is usually done via processing pipelines that consist of a series of individual processing steps that can be executed one at a time. These processing steps are then strung together into workflows and fed with data to address a particular processing goal. Examples of such pipeline processing goals are the quality control of incoming data from telescopes, unsupervised production of science and calibration products for an archive, and supervised data reduction to serve the specific science goals of a scientist. For each of these goals, individual workflows need to be developed. These workflows need to evolve when the pipeline, observing strategies, or calibration plans change. Writing and maintaining such a collection of workflows is therefore a complex and expensive task. Aims. In this paper we propose a data processing system that automatically derives processing workflows for different use cases from a single specification of a cascade of processing steps. Methods. The system works by using formalised descriptions of data processing pipelines that specify the input and output of each processing step. Inputs can be existing data or the output of a previous step. The rules for selecting the most appropriate input data are directly attached to the description. Results. A version of the proposed system has been implemented as the ESO Data Processing System (EDPS) in the Python language. The specification of processing cascades and data organisation rules use a restrictive set of Python classes, attributes, and functions. Conclusions. The EDPS implementation of the proposed system was used to demonstrate that it is possible to automatically derive from a single specification of a pipeline processing cascade the workflows that the European Southern Observatory uses for quality control, archive production, and specialised science reduction. The EDPS will be used to replace all data reduction systems using different workflow specifications that are currently in use at the European Southern Observatory.

Adaptive data reduction workflows for astronomy: The ESO Data Processing System (EDPS)

The data processing pipeline for the MUSE instrument

Reflex: Scientific Workflows for the ESO Pipelines

A fully automated data reduction pipeline for the FRODOSpec integral field spectrograph

Workflow environments for advanced cyberinfrastructure platforms

CRISPRED: A data pipeline for the CRISP imaging spectropolarimeter

A new architecture paradigm for image processing pipeline applied to massive remote sensing data production

exoTEDRF: An EXOplanet Transit and Eclipse Data Reduction Framework

Astronomical Pipeline Provenance: A Use Case Evaluation

General Data Management Workflow to Process Tabular Data in Automated and High throughput Heterogeneous Catalysis Research

Astronomical data organization, management and access in Scientific Data Lakes

From Design to Production Control Through the Integration of Engineering Data Management and Workflow Management Systems

A workflow management system for reproducible and interoperable high-throughput self-driving experiments

Engineering Data Processing Workflows

Data Processing Pipeline for the Earth 2.0 Mission

Automated SpectroPhotometric Image REDuction (ASPIRED)

Adding Workflow Management Flexibility to LSST Pipelines Execution

Streamlining Oceanic Biogeochemical Dataset Assembly in Support of Global Data Products

DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python

The QuickReduce data reduction pipeline for the WIYN One Degree Imager

Improving ALMA's data processing effciency using a holistic approach