Abstract:Motivation: The quality scores data (QSD) account for 70% in compressed FastQ files obtained from the short and long reads sequencing technologies. Designing effective compressors for QSD that counterbalance compression ratio, time cost, and memory consumption is essential in scenarios such as large-scale genomics data sharing and long-term data backup. This study presents a novel parallel lossless QSD-dedicated compression algorithm named PQSDC, which fulfills the above requirements well. PQSDC is based on two core components: a parallel sequences-partition model designed to reduce peak memory consumption and time cost during compression and decompression processes, as well as a parallel four-level run-length prediction mapping model to enhance compression ratio. Besides, the PQSDC algorithm is also designed to be highly concurrent using multicore CPU clusters. Results: We evaluate PQSDC and four state-of-the-art compression algorithms on 27 real-world datasets, including 61.857 billion QSD characters and 632.908 million QSD sequences. (1) For short reads, compared to baselines, the maximum improvement of PQSDC reaches 7.06% in average compression ratio, and 8.01% in weighted average compression ratio. During compression and decompression, the maximum total time savings of PQSDC are 79.96% and 84.56%, respectively; the maximum average memory savings are 68.34% and 77.63%, respectively. (2) For long reads, the maximum improvement of PQSDC reaches 12.51% and 13.42% in average and weighted average compression ratio, respectively. The maximum total time savings during compression and decompression are 53.51% and 72.53%, respectively; the maximum average memory savings are 19.44% and 17.42%, respectively. (3) Furthermore, PQSDC ranks second in compression robustness among the tested algorithms, indicating that it is less affected by the probability distribution of the QSD collections. Overall, our work provides a promising solution for QSD parallel compression, which balances storage cost, time consumption, and memory occupation primely. Availability and implementation: The proposed PQSDC compressor can be downloaded from https://github.com/fahaihi/PQSDC.

A Compression Algorithm of Fastq File Based on Distribution Characteristics Analysis

FastqZip: An Improved Reference-Based Genome Sequence Lossy Compression Framework

PLDSRC: A Multi-threaded Compressor/Decompressor for Massive DNA Sequencing Data

PQSDC: a parallel lossless compressor for quality scores data via sequences partition and run-length prediction mapping

Compressed Data Separation Via ℓq-Split Analysis with ℓ∞-Constraint

Handling the data management needs of high-throughput sequencing data: SpeedGene, a compression algorithm for the efficient storage of genetic data

A Novel Compression Algorithm for High-Throughput DNA Sequence Based on Huffman Coding Method

A new efficient referential genome compression technique for FastQ files

DDQR (dynamic DNA QR coding): An efficient algorithm to represent DNA barcode sequences

FASTA/Q Data Compressors for MapReduce-Hadoop Genomics:Space and Time Savings Made Easy -- Version 1

Analysis of Compression Techniques for DNA Sequence Data

AMGC: Adaptive match-based genomic compression algorithm

GeneSqueeze: A Novel Lossless, Reference-Free Compression Algorithm for FASTQ/A Files

DNA Lossless Differential Compression Algorithm based on Similarity of Genomic Sequence Database

FastqCLS: a FASTQ compressor for long-read sequencing via read reordering using a novel scoring model

CURC: a CUDA-based reference-free read compressor

SFQ: Constructing and Querying a Succinct Representation of FASTQ Files

PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering

GSC: efficient lossless compression of VCF files with fast query

Reference-based genome compression using the longest matched substrings with parallelization consideration

A Novel Lossless Encoding Algorithm for Data Compression - Genomics Data as an Exemplar