Abstract:Context. Astronomical data reduction is usually done via processing pipelines that consist of a series of individual processing steps that can be executed one at a time. These processing steps are then strung together into workflows and fed with data to address a particular processing goal. Examples of such pipeline processing goals are the quality control of incoming data from telescopes, unsupervised production of science and calibration products for an archive, and supervised data reduction to serve the specific science goals of a scientist. For each of these goals, individual workflows need to be developed. These workflows need to evolve when the pipeline, observing strategies, or calibration plans change. Writing and maintaining such a collection of workflows is therefore a complex and expensive task. Aims. In this paper we propose a data processing system that automatically derives processing workflows for different use cases from a single specification of a cascade of processing steps. Methods. The system works by using formalised descriptions of data processing pipelines that specify the input and output of each processing step. Inputs can be existing data or the output of a previous step. The rules for selecting the most appropriate input data are directly attached to the description. Results. A version of the proposed system has been implemented as the ESO Data Processing System (EDPS) in the Python language. The specification of processing cascades and data organisation rules use a restrictive set of Python classes, attributes, and functions. Conclusions. The EDPS implementation of the proposed system was used to demonstrate that it is possible to automatically derive from a single specification of a pipeline processing cascade the workflows that the European Southern Observatory uses for quality control, archive production, and specialised science reduction. The EDPS will be used to replace all data reduction systems using different workflow specifications that are currently in use at the European Southern Observatory.

A fully automated data reduction pipeline for the FRODOSpec integral field spectrograph

Techniques for reducing fiber-fed and integral-field spectroscopy data: The software package R3D

CRISPRED: A data pipeline for the CRISP imaging spectropolarimeter

Data Reduction Pipeline of the TOU Optical Very High Resolution Spectrograph and Its sub-m/s Performance

The QuickReduce data reduction pipeline for the WIYN One Degree Imager

WARP: The Data Reduction Pipeline for the WINERED Spectrograph

Adaptive data reduction workflows for astronomy: The ESO Data Processing System (EDPS)

MASE: A New Data--Reduction Pipeline for the Magellan Echellette Spectrograph

Automated SpectroPhotometric Image REDuction (ASPIRED)

The data processing pipeline for the MUSE instrument

exoTEDRF: An EXOplanet Transit and Eclipse Data Reduction Framework

Robust support for semi-automated reductions of Keck/NIRSPEC data using PypeIt

CARRSSPipeline: Flux Calibration and Non-linear Reprojection for SALT-RSS Multi-Object Spectroscopy over 3500-9500 Å

SIPGI: an interactive pipeline for spectroscopic data reduction

The GIST Pipeline: A Multi-Purpose Tool for the Analysis and Visualisation of (Integral-field) Spectroscopic Data

MEAD: data reduction pipeline for ALES integral field spectrograph and LBTI thermal infrared calibration unit

Pipeline Collector: gathering performance data for distributed astronomical pipelines

A Full Implementation of Spectro-Perfectionism for Precise Radial Velocity Exoplanet Detection: A Test Case With the MINERVA Reduction Pipeline

Pyodine: An open, flexible reduction software for iodine-calibrated precise radial velocities

IO:I: A Near-Infrared Camera for the Liverpool Telescope