Abstract:Suffix Array (SA) is a cardinal data structure in many pattern matching applications, including data compression, plagiarism detection and sequence alignment. However, as the volumes of data increase abruptly, the construction of SA is not amenable to the current large-scale data processing frameworks anymore due to its intrinsic proliferation of suffixes during the construction. That is, ameliorating the performance by just adding the resources to the frameworks becomes less cost- effective, even having the severe diminishing returns. At issue now is whether we can permit SA construction to be more scalable and efficient for the everlasting accretion of data by creating a radical shift in perspective. Regarding TeraSort [1] as our baseline, we first demonstrate the fragile scalability of TeraSort and investigate what causes it through the experiments on the sequence alignment of a grouper (i.e., the SA construc- tion used in bioinformatics). As such, we propose a scheme that amalgamates the distributed key-value store system into MapReduce to leverage the in-memory queries about suffixes. Rather than handling the communication of suffixes, MapReduce is in charge of the communication of their indexes, which means better capacity for more data. It significantly abates the required disk space for constructing SA and better utilizes the memory, which in turn improves the scalability radically. We also examine the efficiency of our scheme in terms of memory and show it outperforms TeraSort. At last, our scheme can complete the pair- end sequencing and alignment with two input files without any degradation on scalability, and can accommodate the suffixes of nearly 6.7 TB in a small cluster composed of 16 nodes and Gigabit Ethernet without any compression.

A new lightweight index SUA for biological sequence analysis

SUA-Based Algorithm for Finding SATRs in DNA Sequence

An Efficient Algorithm for Suffix Sorting

USING AN EXTENDED SUFFIX TREE TO SPEED-UP SEQUENCE ALIGNMENT

Scalable and Efficient Construction of Suffix Array with MapReduce and In-Memory Data Store System

On-line Linear Time Construction of Sequential Binary Suffix Trees

DUHI: Dynamically updated hash index clustering method for DNA storage

A DNA Sequence Alignment Tool Based on BWA and Data Mining

AllSome Sequence Bloom Trees.

Accelerating Sequence Searching: Dimensionality Reduction Method

An Improved Quick Algorithm for Aligning Dna/Rna Sequences

Two simple full-text indexes based on the suffix array

DNA-SaM, a robust system for large-scale data storage

A Bidirectional Fuzzy Index and Approximate Search Algorithm for Next Generation Sequencing

Seeding with Minimized Subsequence.

A Short Reads Alignment Algorithm Oriented To Massive Data

DNA sequences alignment method using sparse index on pan-genome graph

An advanced approach for DNA sequencing and similarities analysis on the basis of groupings of nucleotide bases

Generic Non-Recursive Suffix Array Construction

TUSQ: Targeted High-Utility Sequence Querying

Efficient Seeding for Error-Prone Sequences with SubseqHash2