Abstract:Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement. In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10X faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows.

A Comparative Analysis to the Array Database Technology and Its Use in Flexible VCI Derivation

Research on Decision-making Support System Application for Remote Sensing Integrated Survey of Land Resources

Integration and Management of Massive Remote-Sensing Data Based on GeoSOT Subdivision Model

An Management and Visualization System for Massive Remote Sensing Image: Research and Implementation.

A Study on Data Storage and Management for Massive Remote Sensing Data Based on Multi-level Grid Model

A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

Array Requirements for Scientific Applications and an Implementation for Microsoft SQL Server

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

Method for Managing and Querying Geo-Spatial Data Using a Grid-Code-array Spatial Index

A Tile-Based Framework with a Spatial-Aware Feature for Easy Access and Efficient Analysis of Marine Remote Sensing Data

Research on Database-Based Management Technology for Image Data

Towards A Framework For Offering Remote Sensing Data In An Analysis-Ready Format

A Problem-Oriented Approach for Dem Data Management and Manipulation

Optimized Algorithm Module for Large Volume Remote Sensing Image Processing System

Parallel data analysis directly on scientific file formats

Massive volume data organization in geo-science based on relational database

A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge

A real-time publication system of massive marine remote sensing data - art. no. 67951V

An Efficient Organization Method for Large-Scale and Long Time-Series Remote Sensing Data in a Cloud Computing Environment

Researches on Storage and Access Technologies of Raster Data in Relational Database

SciHive: Array-Based Query Processing with HiveQL