Abstract:Motivation De Bruijn graph, a fundamental data structure to represent and organize genome sequence, plays important roles in various kinds of sequence analysis tasks such as de novo assembly, high-throughput sequencing (HTS) read alignment, pan-genome analysis, metagenomics analysis, HTS read correction, etc. With the rapid development of HTS data and ever-increasing number of assembled genomes, there is a high demand to construct de Bruijn graph for sequences up to Tera-base-pair level. It is non-trivial since the size of the graph to be constructed could be very large and each graph consists of hundreds of billions of vertices and edges. Current existing approaches may have unaffordable memory footprints to handle such a large de Bruijn graph. Moreover, it also requires the construction approach to handle very large dataset efficiently, even if in a relatively small RAM space. Results We propose a lightweight parallel de Bruijn graph construction approach, de Bruijn Graph Constructor in Scalable Memory (deGSM). The main idea of deGSM is to efficiently construct the Bur-rows-Wheeler Transformation (BWT) of the unipaths of de Bruijn graph in constant RAM space and transform the BWT into the original unitigs. It is mainly implemented by a fast parallel external sorting of k-mers, which allows only a part of k-mers kept in RAM by a novel organization of the k-mers. The experimental results demonstrate that, just with a commonly used machine, deGSM is able to handle very large genome sequence(s), e.g., the contigs (305 Gbp) and scaffolds (1.1 Tbp) recorded in Gen-Bank database and Picea abies HTS dataset (9.7 Tbp). Moreover, deGSM also has faster or comparable construction speed compared with state-of-the-art approaches. With its high scalability and efficiency, deGSM has enormous potentials in many large scale genomics studies. Availability <https://github.com/hitbc/deGSM>. Contact ydwang{at}hit.edu.cn (YW) and bo.liu{at}hit.edu.cn (BL) Supplementary information Supplementary data are available online.

Computing the BWT and LCP array of a Set of Strings in External Memory

External memory BWT and LCP computation for sequence collections with applications

A massively parallel algorithm for constructing the BWT of large string sets

The colored longest common prefix array computed via sequential scans

Memory Efficient De Bruijn Graph Construction

Gene Sequence Alignment on a Public Computing Platform

Practical combinations of repetition-aware data structures

Debwt: Parallel Construction of Burrows-Wheeler Transform for Large Collection of Genomes with De Bruijn-branch Encoding.

Processing String Similarity Search in External Memory Efficiently

String Partition for Building Long BWTs

Lightweight Data Indexing and Compression in External Memory

Lightweight merging of compressed indices based on BWT variants

Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform

deGSM: memory scalable construction of large scale de Bruijn Graph

Distributed-Memory Parallel Contig Generation for De Novo Long-Read Genome Assembly

MemLong: Memory-Augmented Retrieval for Long Text Modeling

Genome analysis EPGA 2 : memory-efficient de novo assembler

Sparse Suffix and LCP Array: Simple, Direct, Small, and Fast

Succinct suffix sorting in external memory

Memorize Step by Step: Efficient Long-Context Prefilling with Incremental Memory and Decremental Chunk

String Inference from the LCP Array