Abstract:Suffix Array (SA) is a cardinal data structure in many pattern matching applications, including data compression, plagiarism detection and sequence alignment. However, as the volumes of data increase abruptly, the construction of SA is not amenable to the current large-scale data processing frameworks anymore due to its intrinsic proliferation of suffixes during the construction. That is, ameliorating the performance by just adding the resources to the frameworks becomes less cost- effective, even having the severe diminishing returns. At issue now is whether we can permit SA construction to be more scalable and efficient for the everlasting accretion of data by creating a radical shift in perspective. Regarding TeraSort [1] as our baseline, we first demonstrate the fragile scalability of TeraSort and investigate what causes it through the experiments on the sequence alignment of a grouper (i.e., the SA construc- tion used in bioinformatics). As such, we propose a scheme that amalgamates the distributed key-value store system into MapReduce to leverage the in-memory queries about suffixes. Rather than handling the communication of suffixes, MapReduce is in charge of the communication of their indexes, which means better capacity for more data. It significantly abates the required disk space for constructing SA and better utilizes the memory, which in turn improves the scalability radically. We also examine the efficiency of our scheme in terms of memory and show it outperforms TeraSort. At last, our scheme can complete the pair- end sequencing and alignment with two input files without any degradation on scalability, and can accommodate the suffixes of nearly 6.7 TB in a small cluster composed of 16 nodes and Gigabit Ethernet without any compression.

ERA: Efficient Serial and Parallel Suffix Tree Construction for Very Long Strings

DGST: Efficient and Scalable Suffix Tree Construction on Distributed Data-Parallel Platforms.

A Fast Trie Tree Index Construction Algorithm Using Frequency Characteristic

Scalable and Efficient Construction of Suffix Array with MapReduce and In-Memory Data Store System

Construction of Sparse Suffix Trees and LCE Indexes in Optimal Time and Space

Fast, parallel, and cache-friendly suffix array construction

Relative Suffix Trees

Tunnel: Parallel-inducing sort for large string analytics

Using GPU to Accelerate Suffix Array Construction

Linear Time Online Algorithms for Constructing Linear-size Suffix Trie

Pointer-Machine Algorithms for Fully-Online Construction of Suffix Trees and DAWGs on Multiple Strings

Linear indexing for all strings under all internal nodes in suffix trees

Bass: Approximate Search on Large String Databases

Generic Non-Recursive Suffix Array Construction

Parallel and private generalized suffix tree construction and query on genomic data

An Efficient Word Lookup System by using Improved Trie Algorithm

Trie-Based Data Structures For Sequence Assembly

Linear Index for Logarithmic Search-Time for any String under any Internal Node in Suffix Trees

Sparse Suffix and LCP Array: Simple, Direct, Small, and Fast

Design of Fast Multiple String Searching Based on Improved Prefix Tree

Nucleotide String Indexing using Range Matching