Abstract:The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, through increased stations in the array and frequency span. Two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and I/O. An example of this is the data deluge expected from the SKA Telescopes of more than 60PB per day, all to be stored on the buffer filesystem. Compressing the data is an obvious solution. We used MGARD, an error-controlled compressor, and applied it to simulated and real visibility data, in noise-free and noise-dominated regimes. As the data has an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. Measuring the degradation of images reconstructed using the lossy compressed data, we explore the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images. We found relative error bounds of as much as $10\%$, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a $0.1\%$ error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO. MGARD provides better compression for similar results, and has a host of potentially powerful additional features.

A Lossless Compression Method for Data from a Spaceborne Magnetometer

Lossless Compression of Human Movement IMU Signals.

Lossless compression method for the Magnetic and Helioseismic Imager (MHI) payload

Image compression in local helioseismology

A Novel Method of Lossless Compression for 2-D Astronomical Spectra Images

Polycomp: efficient and configurable compression of astronomical timelines

Lossy Data Compression By Adaptive Mesh Coarsening

Lossless Compression Method for Ultraspectral Sounder Data Based on Key Information Extraction and Spectral–spatial Prediction

Optimising the Processing and Storage of Visibilities using lossy compression

Lossless Astronomical Image Compression and the Effects of Noise

Onboard Lossless Compression of Hyperspectral Imagery Based on Hybrid Prediction

A Method of Lossless Compression for 2D Fiber-Fed Spectra Data

Lossless Compression of Large Aperture Static Imaging Spectrometer Based on CCSDS-123

Hyperspectral IASI L1C Data Compression

Suppression of the Negative Effect of Abnormal Data Based on the Hough Transform and Application to the Magnetic Compensation of Airborne Optically-Pumped Magnetometer Data.

Embedded Transform Coding Based Lossless Compression in Compressive Spectral Imaging with Coded Aperture.

Impact of Lossy Compression Errors on Passive Seismic Data Analyses

Study on the Real-time Lossless Data Compression Method Used in the Readout System for Micro-pattern Gas Detector

Polynomial data compression for large-scale physics experiments

Lossy Compression of Large-Scale Radio Interferometric Data

Lossless and Near-Lossless Compression Algorithms for Remotely Sensed Hyperspectral Images