Abstract:BACKGROUND:The rapid development of Next-Generation Sequencing technologies enables sequencing genomes with low cost. The dramatically increasing amount of sequencing data raised crucial needs for efficient compression algorithms. Reference-based compression algorithms have exhibited outstanding performance on compressing single genomes. However, for the more challenging and more useful problem of compressing a large collection of n genomes, straightforward application of these reference-based algorithms suffers a series of issues such as difficult reference selection and remarkable performance variation.RESULTS:We propose an efficient clustering-based reference selection algorithm for reference-based compression within separate clusters of the n genomes. This method clusters the genomes into subsets of highly similar genomes using MinHash sketch distance, and uses the centroid sequence of each cluster as the reference genome for an outstanding reference-based compression of the remaining genomes in each cluster. A final reference is then selected from these reference genomes for the compression of the remaining reference genomes. Our method significantly improved the performance of the-state-of-art compression algorithms on large-scale human and rice genome databases containing thousands of genome sequences. The compression ratio gain can reach up to 20-30% in most cases for the datasets from NCBI, the 1000 Human Genomes Project and the 3000 Rice Genomes Project. The best improvement boosts the performance from 351.74 compression folds to 443.51 folds.CONCLUSIONS:The compression ratio of reference-based compression on large scale genome datasets can be improved via reference selection by applying appropriate data preprocessing and clustering methods. Our algorithm provides an efficient way to compress large genome database.

RabbitTClust: Enabling Fast Clustering Analysis of Millions of Bacteria Genomes with MinHash Sketches.

Two-Stage Clustering (Tsc): A Pipeline For Selecting Operational Taxonomic Units For The High-Throughput Sequencing Of Pcr Amplicons

RabbitKSSD: Accelerating Genome Distance Estimation on Modern Multi-Core Architectures.

Gclust:A Parallel Clustering Tool for Microbial Genomic Data

RabbitMash: Accelerating Hash-Based Genome Analysis on Modern Multi-Core Architectures.

RabbitV: Fast Detection of Viruses and Microorganisms in Sequencing Data on Multi-Core Architectures

Sketch Distance-Based Clustering of Chromosomes for Large Genome Database Compression.

RabbitQCPlus: More Efficient Quality Control for Sequencing Data.

RabbitQCPlus 2.0: More efficient and versatile quality control for sequencing data

RabbitFX: Efficient Framework for FASTA/Q File Parsing on Modern Multi-Core Platforms.

Fractional Hitting Sets for Efficient and Lightweight Genomic Data Sketching

Clustering one million molecular structures on GPU within seconds

RabbitQC: high-speed scalable quality control for sequencing data

RabbitTrim: Highly Optimized Trimming of Illumina Sequencing Data on Multi-core Platforms

RabbitSAlign: Accelerating Short-Read Alignment for CPU-GPU Heterogeneous Platforms

A Novel Clustering Method Via Nucleotide-Based Fourier Power Spectrum Analysis

Efficient clustering of large molecular libraries

Gcluster: a simple-to-use tool for visualizing and comparing genome contexts for numerous genomes.

SSCC: a novel computational framework for rapid and accurate clustering large single cell RNA-seq data

Interpolative Multidimensional Scaling Techniques for the Identification of Clusters in Very Large Sequence Sets

Highly efficient clustering of long-read transcriptomic data with GeLuster