Abstract:In the era of petascale computing, more scientific applications are being deployed on leadership scale computing platforms to enhance the scientific productivity. Many I/O techniques have been designed to address the growing I/O bottleneck on large-scale systems by handling massive scientific data in a holistic manner. While such techniques have been leveraged in a wide range of applications, they have not been shown as adequate for many mission critical applications, particularly in data post-processing stage. One of the examples is that some scientific applications generate datasets composed of a vast amount of small data elements that are organized along many spatial and temporal dimensions but require sophisticated data analytics on one or more dimensions. Including such dimensional knowledge into data organization can be beneficial to the efficiency of data post-processing, which is often missing from exiting I/O techniques. In this study, we propose a novel I/O scheme named STAR ( Spatial and Temporal AggRegation) to enable high performance data queries for scientific analytics. STAR is able to dive into the massive data, identify the spatial and temporal relationships among data variables, and accordingly organize them into an optimized multi-dimensional data structure before storing to the storage. This technique not only facilitates the common access patterns of data analytics, but also further reduces the application turnaround time. In particular, STAR is able to enable efficient data queries along the time dimension, a practice common in scientific analytics but not yet supported by existing I/O techniques. In our case study with a critical climate modeling application GEOS-5, the experimental results on Jaguar supercomputer demonstrate an improvement up to 73 times for the read performance compared to the original I/O method.

SciAP: A Programmable, High-Performance Platform for Large-Scale Scientific Data

SCAN: A Smart Application Platform for Empowering Parallelizations of Big Genomic Data Analysis in Clouds

Sdac: Porting Scientific Data To Spark Rdds

Distributed data analysis and processing platform based on Pig_Spark

ParSA: High-throughput Scientific Data Analysis Framework with Distributed File System

A Benchmarking Study to Evaluate Apache Spark on Large-Scale Supercomputers

Design and implementation of reconfigurable acceleration for in-memory distributed big data computing.

A Lightweight I/O Scheme to Facilitate Spatial and Temporal Queries of Scientific Data Analytics.

An Experimental Study of a Biosequence Big Data Analysis Service

Application-Aware Storage Strategy For Scientific Data

Introduction to Harp: when Big Data Meets HPC

Accelerating Large-Scale Genomic Analysis With Spark

Framing Apache Spark in life sciences

In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model

Combining HPC and Big Data Infrastructures in Large-Scale Post-Processing of Simulation Data: A Case Study.

BDAP: A Big Data Analysis Platform Based on Spark

Swmpas-A: Scaling MPAS-A to 39 Million Heterogeneous Cores on the New Generation Sunway Supercomputer

SciHive: Array-Based Query Processing with HiveQL

Octopus-DF: Unified DataFrame-based cross-platform data analytic system

A Case Study of Accelerating Apache Spark with FPGA.

Bioinformatics Applications on Apache Spark