Abstract:The data volumes stored in telescope archives is constantly increasing due to the development and improvements in the instrumentation. Often the archives need to be stored over a distributed storage architecture, provided by independent compute centres. Such a distributed data archive requires overarching data management orchestration. Such orchestration comprises of tools which handle data storage and cataloguing, and steering transfers integrating different storage systems and protocols, while being aware of data policies and locality. In addition, it needs a common Authorisation and Authentication Infrastructure (AAI) layer which is perceived as a single entity by end users and provides transparent data access. The scientific domain of particle physics also uses complex and distributed data management systems. The experiments at the Large Hadron Collider\,(LHC) accelerator at CERN generate several hundred petabytes of data per year. This data is globally distributed to partner sites and users using national compute facilities. Several innovative tools were developed to successfully address the distributed computing challenges in the context of the Worldwide LHC Computing Grid (WLCG). The work being carried out in the ESCAPE project and in the Data Infrastructure for Open Science (DIOS) work package is to prototype a Scientific Data Lake using the tools developed in the context of the WLCG, harnessing different physics scientific disciplines addressing FAIR standards and Open Data. We present how the Scientific Data Lake prototype is applied to address astronomical data use cases. We introduce the software stack and also discuss some of the differences between the domains.

Data Lakes for Digital Humanities

On data lake architectures and metadata management

ArchaeoDAL: A Data Lake for Archaeological Data Management and Analytics

Semantic Data Management in Data Lakes

Toward data lakes as central building blocks for data management and analysis

Data Lakes: A Survey of Functions and Systems

Spatial big data architecture: From Data Warehouses and Data Lakes to the LakeHouse

A Zone-Based Data Lake Architecture for IoT, Small and Big Data

Benchmarking Data Lakes Featuring Structured and Unstructured Data with DLBench

Data Lake Ingestion Management

DLToDW: Transferring Relational and NoSQL Databases from a Data Lake

Dataversifying Natural Sciences: Pioneering a Data Lake Architecture for Curated Data-Centric Experiments in Life \& Earth Sciences

Metadata Systems for Data Lakes: Models and Features

Analytic Processing in Data Lakes: A Semantic Query-Driven Discovery Approach

Joint Management and Analysis of Textual Documents and Tabular Data within the AUDAL Data Lake

Astronomical data organization, management and access in Scientific Data Lakes

DatAasee -- A Metadata-Lake as Metadata Catalog for a Virtual Data-Lake

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Data Lakehouse: Next Generation Information System

The Data Lakehouse: Data Warehousing and More

Manipulating Data Lakes Intelligently With Java Annotations