Abstract:We propose a new technique for creating a space-efficient index for large repetitive text collections, such as pangenomic databases containing sequences of many individuals from the same species. We combine two recent techniques from this area: Wheeler graphs (Gagie et al., 2017) and prefix-free parsing (PFP, Boucher et al., 2019). Wheeler graphs (WGs) are a general framework encompassing several indexes based on the Burrows-Wheeler transform (BWT), such as the FM-index. Wheeler graphs admit a succinct representation which can be further compacted by employing the idea of tunnelling, which exploits redundancies in the form of parallel, equally-labelled paths called blocks that can be merged into a single path. The problem of finding the optimal set of blocks for tunnelling, i.e. the one that minimizes the size of the resulting WG, is known to be NP-complete and remains the most computationally challenging part of the tunnelling process. To find an adequate set of blocks in less time, we propose a new method based on the prefix-free parsing (PFP). The idea of PFP is to divide the input text into phrases of roughly equal sizes that overlap by a fixed number of characters. The original text is represented by a sequence of phrase ranks (the parse) and a list of all used phrases (the dictionary). In repetitive texts, the PFP of the text is generally much shorter than the original. To speed up the block selection for tunnelling, we apply the PFP to obtain the parse and the dictionary of the text, tunnel the WG of the parse using existing heuristics and subsequently use this tunnelled parse to construct a compact WG of the original text. Compared with constructing a WG from the original text without PFP, our method is much faster and uses less memory on collections of pangenomic sequences. Therefore, our method enables the use of WGs as a pangenomic reference for real-world datasets.

String Partition for Building Long BWTs

Efficient construction of the extended BWT from grammar-compressed DNA sequencing reads

Debwt: Parallel Construction of Burrows-Wheeler Transform for Large Collection of Genomes with De Bruijn-branch Encoding.

BWT construction and search at the terabase scale

A massively parallel algorithm for constructing the BWT of large string sets

Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform

BWTCP: A Parallel Method for Constructing BWT in Large Collection of Genomic Reads.

A Novel Hardware Architecture to Accelerate Burrows-Wheeler Transform

A survey of BWT variants for string collections

Practical combinations of repetition-aware data structures

GPU-Accelerated BWT Construction for Large Collection of Short Reads

A natural encoding of genetic variation in a Burrows-Wheeler Transform to enable mapping and genome inference

Computing the BWT and LCP array of a Set of Strings in External Memory

Simulating the DNA String Graph in Succinct Space

Prefix-free parsing for building large tunnelled Wheeler graphs

LinearPartition: linear-time approximation of RNA folding partition function and base-pairing probabilities

Memory-Aware BWT by segmenting sequences to support subsequence search

Efficient Parallel Design for BWT-based DNA Sequences Data Multi-Compression Algorithm

Ultrafast and memory-efficient alignment of short DNA sequences to the human genome

External memory BWT and LCP computation for sequence collections with applications

Memory Efficient De Bruijn Graph Construction