Abstract:Many scientific applications opt for particles instead of meshes as their basic primitives to model complex systems composed of billions of discrete entities. Such applications span a diverse array of scientific domains, including molecular dynamics, cosmology, computational fluid dynamics, and geology. The scale of the particles in those scientific applications increases substantially thanks to the ever-increasing computational power in high-performance computing (HPC) platforms. However, the actual gains from such increases are often undercut by obstacles in data management systems related to data storage, transfer, and processing. Lossy compression has been widely recognized as a promising solution to enhance scientific data management systems regarding such challenges, although most existing compression solutions are tailored for Cartesian grids and thus have sub-optimal results on discrete particle data. In this paper, we introduce LCP, an innovative lossy compressor designed for particle datasets, offering superior compression quality and higher speed than existing compression solutions. Specifically, our contribution is threefold. (1) We propose LCP-S, an error-bound aware block-wise spatial compressor to efficiently reduce particle data size. This approach is universally applicable to particle data across various domains. (2) We develop LCP, a hybrid compression solution for multi-frame particle data, featuring dynamic method selection and parameter optimization. (3) We evaluate our solution alongside eight state-of-the-art alternatives on eight real-world particle datasets from seven distinct domains. The results demonstrate that our solution achieves up to 104% improvement in compression ratios and up to 593% increase in speed compared to the second-best option, under the same error criteria.

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

TAC: Optimizing Error-Bounded Lossy Compression for Three-Dimensional Adaptive Mesh Refinement Simulations

AMRIC: A Novel In Situ Lossy Compression Framework for Efficient I/O in Adaptive Mesh Refinement Applications

TAC+: Optimizing Error-Bounded Lossy Compression for 3D AMR Simulations

Adaptive Decomposition and Shared Weight Volumetric Transformer Blocks for Efficient Patch-Free 3D Medical Image Segmentation.

Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications

Accelerating In-transit Isosurface Generation With Topology Preserving Compression

Lossy Data Compression By Adaptive Mesh Coarsening

Compressing Repeated Content Within Large-Scale Remote Sensing Images

Surface Carving-Based Automatic Volume Data Reduction

Accelerating Lossy Compression on HPC Datasets Via Partitioning Computation for Parallel Processing

Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds

LCP: Enhancing Scientific Data Management with Lossy Compression for Particles

Accelerating Relative-error Bounded Lossy Compression for HPC Datasets with Precomputation-Based Mechanisms.

Artifact-Minimized High-Ratio Image Compression with Preserved Analysis Fidelity

HexaShrink, an exact scalable framework for hexahedral meshes with attributes and discontinuities: multiresolution rendering and storage of geoscience models

Scalable Hybrid Learning Techniques for Scientific Data Compression

Improving I/O Throughput of Scientific Applications Using Transparent Parallel Compression

LAMP: Improving Compression Ratio for AMR Applications Via Level Associated Mapping-Based Preconditioning

A framework for compressing unstructured scientific data via serialization

Total Variation Reduction for Lossless Compression of HPC Applications