Abstract:Access libraries such as ROOT and HDF5 allow users to interact with datasets using high level abstractions, like coordinate systems and associated slicing operations. Unfortunately, the implementations of access libraries are based on outdated assumptions about storage systems interfaces and are generally unable to fully benefit from modern fast storage devices. The situation is getting worse with rapidly evolving storage devices such as non-volatile memory and ever larger datasets. This project explores distributed dataset mapping infrastructures that can integrate and scale out existing access libraries using Ceph's extensible object model, avoiding re-implementation or even modifications of these access libraries as much as possible. These programmable storage extensions coupled with our distributed dataset mapping techniques enable: 1) access library operations to be offloaded to storage system servers, 2) the independent evolution of access libraries and storage systems and 3) fully leveraging of the existing load balancing, elasticity, and failure management of distributed storage systems like Ceph. They also create more opportunities to conduct storage server-local optimizations specific to storage servers. For example, storage servers might include local key/value stores combined with chunk stores that require different optimizations than a local file system. As storage servers evolve to support new storage devices like non-volatile memory, these server-local optimizations can be implemented while minimizing disruptions to applications. We will report progress on the means by which distributed dataset mapping can be abstracted over particular access libraries, including access libraries for ROOT data, and how we address some of the challenges revolving around data partitioning and composability of access operations.

Bixi: A EB-level Object Storage System Based on CEPH.

A Request Skew Aware Heterogeneous Distributed Storage System Based on Cassandra

Indexing multi-dimensional data in a cloud system.

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

BlobSeer: How to Enable Efficient Versioning for Large Object Storage under Heavy Access Concurrency

Towards Cluster-wide Deduplication Based on Ceph

A Ceph-based storage strategy for big gridded remote sensing data

Atlas: Baidu's key-value storage system for cloud data

ES2: A cloud data storage system for supporting both OLTP and OLAP

A New Approach to Double I/O Performance for Ceph Distributed File System in Cloud Computing

Rhobbs: An Enhanced Hybrid Storage Providing Block Storage For Virtual Machines

E2FS: an Elastic Storage System for Cloud Computing

x Meta : SSD-HDD-Hybrid Optimization for Metadata Maintenance of Cloud-Scale Object Storage

Providing Scalable Database Services On The Cloud

A Ceph S3 Object Data Store for HEP

Achelous: Enabling Programmability, Elasticity, and Reliability in Hyperscale Cloud Networks.

Elastic HDFS: interconnected distributed architecture for availability–scalability enhancement of large-scale cloud storages

Mapping Datasets to Object Storage System

An Efficient and Scalable Multi-Dimensional Indexing Scheme for Modular Data Centers.

HyCloud: Tweaking Hybrid Cloud Storage Services for Cost-Efficient Filesystem Hosting

Survey the storage systems used in HPC and BDA ecosystems