Abstract:Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies impose strict precision requirements, the impact of such lossy data compression on the 21-cm signal power spectrum statistic should be understood. We apply Dysco visibility compression, a technique to normalize and quantize specifically designed for radio interferometric data. We establish the level of the compression noise in the power spectrum in comparison to the thermal noise as well as its coherency behavior. Finally, for optimal compression results, we compare the compression noise obtained from different compression settings to a nominal 21-cm signal power. From a single night of observation, we find that the noise introduced due to the compression is more than five orders of magnitude lower than the thermal noise level in the power spectrum. The noise does not affect calibration. The compression noise shows no correlation with the sky signal and has no measurable coherent component. The level of compression error in the power spectrum ultimately depends on the compression settings. Dysco visibility compression is found to be of insignificant concern for 21-cm power spectrum studies. Hence, data volumes can be safely reduced by factors of $\sim 4$ and with insignificant bias to the final power spectrum. Data from SKA-low will likely be compressible by the same factor as LOFAR, owing to the similarities of the two instruments. The same technique can be used to compress data from other telescopes, but a small adjustment of the compression parameters might be required.

Image compression in local helioseismology

Optimising the Processing and Storage of Visibilities using lossy compression

Lossless Compression of Human Movement IMU Signals.

Impact of Lossy Compression Errors on Passive Seismic Data Analyses

Lossless Astronomical Image Compression and the Effects of Noise

The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR

Lossy Compression of Large-Scale Radio Interferometric Data

Lossless compression method for the Magnetic and Helioseismic Imager (MHI) payload

Neural-based Compression Scheme for Solar Image Data

Local helioseismology and correlation tracking analysis of surface structures in realistic simulations of solar convection

Polycomp: efficient and configurable compression of astronomical timelines

In-Depth Exploration of Single-Snapshot Lossy Compression Techniques for N-Body Simulations

Lossless and Near-Lossless Compression Algorithms for Remotely Sensed Hyperspectral Images

Enabling near real-time remote search for fast transient events with lossy data compression

Testing Helioseismic-Holography Inversions for Supergranular Flows Using Synthetic Data

Compressing Repeated Content Within Large-Scale Remote Sensing Images

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

Scalable Hybrid Learning Techniques for Scientific Data Compression

Lossy Image Compression in a Preclinical Multimodal Imaging Study

Deep data compression for approximate ultrasonic image formation

Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information