Abstract:Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement. In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10X faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows.

The A4 project: physics data processing using the Google protocol buffer library

Building A High Performance Parallel File System Using Grid Datafarm and ROOT I/O

Astronomical Data Preprocessing Implementation Based On Fpga And Data Transformation Strategy For The Fast Telescope As A Giant Cps

An Operator Library System for HEP Events Processing Based on Hbase Coprocessor.

Toward real-time data query systems in HEP

High-Performance I/O: HDF5 for Lattice QCD

Geant4: a Game Changer in High Energy Physics and Related Applicative Fields

Research on General Data Format and I/O Interface for High Power Microwave Numerical Simulations

Parallel data analysis directly on scientific file formats

Using Big Data Technologies for HEP Analysis

REST-for-Physics, a ROOT-based Framework for Event Oriented Data Analysis and Combined Monte Carlo Response

A customizable data management framework for high-repetition-rate high-energy-density science

A redistribution tool for long-term archive of astronomical observation data

Objectivity Data Migration

AIP: a Tool for Flexible and Transparent Data Management

Data Management Challenges and Event Index Technologies in High Energy Physics

S4: an O2r/R2o Infrastructure for Optimizing Satellite Data Utilization in Noaa Numerical Modeling Systems A Step Toward Bridging the Gap Between Research and Operations

Recent developments in Geant4

JADLib-A High Level I/O Library for Scientific Computing Data

Evolution of the ROOT Tree I/O