Abstract:Access libraries such as ROOT and HDF5 allow users to interact with datasets using high level abstractions, like coordinate systems and associated slicing operations. Unfortunately, the implementations of access libraries are based on outdated assumptions about storage systems interfaces and are generally unable to fully benefit from modern fast storage devices. The situation is getting worse with rapidly evolving storage devices such as non-volatile memory and ever larger datasets. This project explores distributed dataset mapping infrastructures that can integrate and scale out existing access libraries using Ceph's extensible object model, avoiding re-implementation or even modifications of these access libraries as much as possible. These programmable storage extensions coupled with our distributed dataset mapping techniques enable: 1) access library operations to be offloaded to storage system servers, 2) the independent evolution of access libraries and storage systems and 3) fully leveraging of the existing load balancing, elasticity, and failure management of distributed storage systems like Ceph. They also create more opportunities to conduct storage server-local optimizations specific to storage servers. For example, storage servers might include local key/value stores combined with chunk stores that require different optimizations than a local file system. As storage servers evolve to support new storage devices like non-volatile memory, these server-local optimizations can be implemented while minimizing disruptions to applications. We will report progress on the means by which distributed dataset mapping can be abstracted over particular access libraries, including access libraries for ROOT data, and how we address some of the challenges revolving around data partitioning and composability of access operations.

Objectivity Data Migration

CASTOR status and evolution

Integrated Data Acquisition, Storage, Retrieval and Processing Using the COMPASS DataBase (CDB)

Persistent storage of non-event data in the CMS databases

Database independent Migration of Objects into an Object-Relational Database

A customizable data management framework for high-repetition-rate high-energy-density science

Building A High Performance Parallel File System Using Grid Datafarm and ROOT I/O

Scalable Database Access Technologies for ATLAS Distributed Computing

Automated, Reliable, and Efficient Continental-Scale Replication of 7.3 Petabytes of Climate Simulation Data: A Case Study

Grid Collector: an Event Catalog with Automated File Management

LHC Databases on the Grid: Achievements and Open Issues

Quantum spiral bandwidth of entangled two-photon states

Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations

Efficient and Reliable Data Management for Biomedical Applications

Mapping Datasets to Object Storage System

Dynamo -- Handling Scientific Data Across Sites and Storage Media

A distributed data warehouse system for astroparticle physics

Efficient Multi-site Data Movement Using Constraint Programming for Data Hungry Science

Astronomical data organization, management and access in Scientific Data Lakes

Federating distributed storage for clouds in ATLAS

Using Big Data Technologies for HEP Analysis