Abstract:Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement. In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10X faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows.

Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers

Accelerating Time-to-Science by Streaming Detector Data Directly into Perlmutter Compute Nodes

End-to-end Online Performance Data Capture and Analysis for Scientific Workflows

Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers

Employing artificial intelligence to steer exascale workflows with colmena

Towards In-transit Analysis on Supercomputing Environments

Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing

Scalable Deployment of a LIGO Physics Application on Public Clouds: Workflow Engine and Resource Provisioning Techniques

A Survey of Data-Intensive Scientific Workflow Management

Big Science with a Small Budget: Non-Embarrassingly Parallel Applications in a Non-Dedicated Network of Workstations

SCISPACE: A Scientific Collaboration Workspace for File Systems in Geo-Distributed HPC Data Centers

Reproducible and Portable Workflows for Scientific Computing and HPC in the Cloud

Improving Scientific Workflow with Cloud Offloading

MeDiCi-Cloud: A Workflow Infrastructure for Large-scale Scientific Applications

Toward a Dynamic Network-Centric Distributed Cloud Platform for Scientific Workflows: A Case Study for Adaptive Weather Sensing

Parallel data analysis directly on scientific file formats

Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments

Accelerating Large Scale Scientific Exploration through Data Diffusion

SOMA: Observability, monitoring, and in situ analytics for exascale applications

Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations