Abstract:The data volumes stored in telescope archives is constantly increasing due to the development and improvements in the instrumentation. Often the archives need to be stored over a distributed storage architecture, provided by independent compute centres. Such a distributed data archive requires overarching data management orchestration. Such orchestration comprises of tools which handle data storage and cataloguing, and steering transfers integrating different storage systems and protocols, while being aware of data policies and locality. In addition, it needs a common Authorisation and Authentication Infrastructure (AAI) layer which is perceived as a single entity by end users and provides transparent data access. The scientific domain of particle physics also uses complex and distributed data management systems. The experiments at the Large Hadron Collider\,(LHC) accelerator at CERN generate several hundred petabytes of data per year. This data is globally distributed to partner sites and users using national compute facilities. Several innovative tools were developed to successfully address the distributed computing challenges in the context of the Worldwide LHC Computing Grid (WLCG). The work being carried out in the ESCAPE project and in the Data Infrastructure for Open Science (DIOS) work package is to prototype a Scientific Data Lake using the tools developed in the context of the WLCG, harnessing different physics scientific disciplines addressing FAIR standards and Open Data. We present how the Scientific Data Lake prototype is applied to address astronomical data use cases. We introduce the software stack and also discuss some of the differences between the domains.

The International Lattice Data Grid -- towards FAIR Data

The International Lattice Data Grid (ILDG 2.0)

Making use of the International Lattice Data Grid

KEK GRID for ILC Experiments

Open Science in Lattice Gauge Theory community

A metadata schema for lattice thermal conductivity from first-principles calculations

FAIR Data Cube, a FAIR data infrastructure for integrated multi-omics data analysis

Building a Disciplinary, World-Wide Data Infrastructure

Shared Metadata for Data-Centric Materials Science

The International Large Detector (ILD) for a future electron-positron collider: Status and Plans

High-Performance I/O: HDF5 for Lattice QCD

Data Life Cycle Labs, A New Concept to Support Data-Intensive Science

The Challenges of Linked Open Data Semantic Enrichment, Discovery, and Dissemination (GRID)

FAIR high level data for Cherenkov astronomy

nanoHUB services for FAIR simulations and data: ResultsDB and Sim2Ls

A Grid Middleware DISQ for Data Integration

From Theory to Practice: Demonstrators of FAIR Data Spaces Across Different Sectors

Astronomical data organization, management and access in Scientific Data Lakes

Lattice studies of Sp(2N) gauge theories using GRID

Grid-enabled Standard-compliant Open Computing Environment for Earth Science Exploration and Applications

Community Action on FAIR Data will Fuel a Revolution in Materials Research