Abstract:The suffix tree is a fundamental data structure for string processing. It is widely used in many important scenarios such as text processing, information retrieval, and bioinformatics. With the rapid growth of data volume, constructing the suffix tree for large-scale datasets is very time-consuming. To solve this problem, a number of MPI-based parallel algorithms were proposed, but they have limitations in fault tolerance and scalability for large-scale datasets. Recently, there are ever-increasing application demands on efficient algorithms for constructing the suffix tree for large-scale datasets on distributed data-parallel platforms, such as Hadoop and Spark. In this paper, we present DGST, which is an efficient and scalable algorithm for generalized suffix tree construction on distributed data-parallel platforms. DGST consists of two major stages: parallel sub-tree partitioning and parallel sub-tree construction. We first design a novel data partitioning strategy for both two stages in the data-parallel paradigm. Then, we propose an efficient sub-tree partitioning algorithm based on parallel frequency counting. To improve the load balance and amortize the disk I/O costs, we propose an efficient Bin-Packing and Number-Partitioning based task allocation strategy for the sub-tree construction. At the sub-tree construction stage, we further propose a novel data structure LCP-Range and a multi-way LCP-Merge sorting algorithm for parallel LCP array construction. The experimental results on Apache Spark reveal that DGST outperforms the state-of-the-art ERa algorithm with approximately 3 times speedup on both the DNA and English text datasets. Furthermore, DGST achieves near-linear data and node scalability. (C) 2019 Elsevier B.V. All rights reserved.

DGCF: A Distributed Greedy Clustering Framework for Large-scale Genomic Sequences.

Gene Sequence Alignment on a Public Computing Platform

Gclust:A Parallel Clustering Tool for Microbial Genomic Data

Dace: A Scalable Dp-Means Algorithm for Clustering Extremely Large Sequence Data

An Efficient Greedy Incremental Sequence Clustering Algorithm

Ngia: A Novel Greedy Incremental Alignment Based Algorithm for Gene Sequence Clustering

Large-Scale DNA Sequence Assembly by Using Computing Grid

CD-HIT: accelerated for clustering the next-generation sequencing data

Robust and Rapid Algorithms Facilitate Large-Scale Whole Genome Sequencing Downstream Analysis in an Integrative Framework

Accelerating Gene Clustering on Heterogeneous Clusters

Simplified DGS Procedure for Large-Scale Genome Structural Study

Parallel Local Alignment Algorithm for Multiple Sequences on Heterogeneous Cluster Systems

Distributed-Memory Parallel Contig Generation for De Novo Long-Read Genome Assembly

Accelerating Alignment for Short Reads Allowing Insertion of Gaps on Multi-Core Cluster

Parallel Algorithms for Large-Scale Biological Sequence Alignment on Xeon-Phi Based Clusters

DGST: Efficient and Scalable Suffix Tree Construction on Distributed Data-Parallel Platforms.

DDP-B - A Distributed Dynamic Parallel Framework for Meta-genomics Binary Similarity.

Distributed structural clustering on large graph

Interpolative Multidimensional Scaling Techniques for the Identification of Clusters in Very Large Sequence Sets

Computational Strategies for Scalable Genomics Analysis

Efficient Distributed Parallel Aligning Reads and Reference Genome with Many Repetitive Subsequences Using Compact de Bruijn Graph