Abstract:CONTRIBUTIONS:We developed a new lossless compression method for WIG data, named smallWig, offering the best known compression rates for RNA-seq data and featuring random access functionalities that enable visualization, summary statistics analysis and fast queries from the compressed files. Our approach results in order of magnitude improvements compared with bigWig and ensures compression rates only a fraction of those produced by cWig. The key features of the smallWig algorithm are statistical data analysis and a combination of source coding methods that ensure high flexibility and make the algorithm suitable for different applications. Furthermore, for general-purpose file compression, the compression rate of smallWig approaches the empirical entropy of the tested WIG data. For compression with random query features, smallWig uses a simple block-based compression scheme that introduces only a minor overhead in the compression rate. For archival or storage space-sensitive applications, the method relies on context mixing techniques that lead to further improvements of the compression rate. Implementations of smallWig can be executed in parallel on different sets of chromosomes using multiple processors, thereby enabling desirable scaling for future transcriptome Big Data platforms.MOTIVATION:The development of next-generation sequencing technologies has led to a dramatic decrease in the cost of DNA/RNA sequencing and expression profiling. RNA-seq has emerged as an important and inexpensive technology that provides information about whole transcriptomes of various species and organisms, as well as different organs and cellular communities. The vast volume of data generated by RNA-seq experiments has significantly increased data storage costs and communication bandwidth requirements. Current compression tools for RNA-seq data such as bigWig and cWig either use general-purpose compressors (gzip) or suboptimal compression schemes that leave significant room for improvement. To substantiate this claim, we performed a statistical analysis of expression data in different transform domains and developed accompanying entropy coding methods that bridge the gap between theoretical and practical WIG file compression rates.RESULTS:We tested different variants of the smallWig compression algorithm on a number of integer-and real- (floating point) valued RNA-seq WIG files generated by the ENCODE project. The results reveal that, on average, smallWig offers 18-fold compression rate improvements, up to 2.5-fold compression time improvements, and 1.5-fold decompression time improvements when compared with bigWig. On the tested files, the memory usage of the algorithm never exceeded 90 KB. When more elaborate context mixing compressors were used within smallWig, the obtained compression rates were as much as 23 times better than those of bigWig. For smallWig used in the random query mode, which also supports retrieval of the summary statistics, an overhead in the compression rate of roughly 3-17% was introduced depending on the chosen system parameters. An increase in encoding and decoding time of 30% and 55% represents an additional performance loss caused by enabling random data access. We also implemented smallWig using multi-processor programming. This parallelization feature decreases the encoding delay 2-3.4 times compared with that of a single-processor implementation, with the number of processors used ranging from 2 to 8; in the same parameter regime, the decoding delay decreased 2-5.2 times.AVAILABILITY AND IMPLEMENTATION:The smallWig software can be downloaded from: http://stanford.edu/~zhiyingw/smallWig/smallwig.html, http://publish.illinois.edu/milenkovic/, http://web.stanford.edu/~tsachy/.CONTACT:zhiyingw@stanford.eduSUPPLEMENTARY INFORMATION:Supplementary data are available at Bioinformatics online.

A fast machine learning dataloader for epigenetic tracks from BigWig files

Megadepth: efficient coverage quantification for BigWigs and BAMs

Smallwig: Parallel Compression of Rna-Seq Wig Files

ChIPWig: A Random Access-Enabling Lossless and Lossy Compression Method for ChIP-seq Data

CoGe LoadExp+: A web‐based suite that integrates next‐generation sequencing data analysis workflows and visualization

Efficient storage and regression computation for population-scale genome sequencing studies

Computing linkage disequilibrium aware genome embeddings using autoencoders

Scywalker: Scalable end-to-end data analysis workflow for long-read single-cell transcriptome sequencing

Pybedgraph: a Python Package for Fast Operations on 1D Genomic Signal Tracks

Scywalker: scalable end-to-end data analysis workflow for nanopore single-cell transcriptome sequencing

Bigtools: a high-performance BigWig and BigBed library in Rust

SPANDx: a genomics pipeline for comparative analysis of large haploid whole genome re-sequencing datasets

wgatools: an ultrafast toolkit for manipulating whole genome alignments

Metapipeline-DNA: A Comprehensive Germline & Somatic Genomics Nextflow Pipeline

wgbstools: A computational suite for DNA methylation sequencing data representation, visualization, and analysis

A Robust Parallel Computing Data Extraction Framework for Nanopore Experiments

Massive Genomic Data Processing and Deep Analysis

Ecdna Machine Learning Modeling

DeepGenGrep: a general deep learning-based predictor for multiple genomic signals and regions

A Fast Word Embedding Based Classifier to Profile Target Gene Databases in Metagenomic Samples

aPEAch: Automated Pipeline for End-to-End Analysis of Epigenomic and Transcriptomic Data