Abstract:The growing velocity of biological big data is way beyond Moore's Law of compute power growth. The amount of genomic data has been explosively accumulating, which calls for an enormous amount of computing power, while current computation methods cannot scale out with the data explosion. In this paper, we try to utilize huge computing resources to solve thebig dataproblems of genome processing on TH-2 supercomputer. TH-2supercomputer adopts neo-heterogeneous architecture and owns 16,000 compute nodes: 32000 Intel Xeon CPUs + 48000 Xeon Phi MICs. The heterogeneity, scalability, and parallel efficiency pose great challenges forthe deployment of the genomeanalysis software pipeline on TH-2. Runtime profiling shows that SOAP3-dp and SOAPsnp are the most time-consuming parts up to 70% of total runtime in the whole pipeline, which need parallelized optimization deeply and large-scale deployment. To address this issue, we first designa series of new parallel algorithms for SOAP3-dp and SOAPsnp, respectively, to eliminatethe spatial-temporal redundancy. Then we propose a CPU/MIC collaboratedparallel computing method in one node to fully fill the CPU/MIC time slots. We also propose a series ofscalable parallel algorithms and large scaleprogramming methods to reduce the amount of communications between different nodes. Moreover, we deploy and evaluate our works on the TH-2 supercomputer in different scales. At the most large scale, the whole process takes 8.37 hours using 8192 nodes to finish the analysis of a 300TB dataset of whole genome sequences from 2,000 human beings, which can take as long as 8 months on a commodity server. The speedup is about 700x.

Construction and Application of a Large-Scale DNA Sequence Analysis System Based on PC Linux

Construction and Application of an in Silico Elongation System of Nucleic Acid Sequence

SaAlign: Multiple DNA/RNA Sequence Alignment and Phylogenetic Tree Construction Tool for Ultra-Large Datasets and Ultra-Long Sequences Based on Suffix Array

Analyzing large-scale DNA Sequences on Multi-core Architectures

Neo-hetergeneous Programming and Parallelized Optimization of a Human Genome Re-sequencing Analysis Software Pipeline on TH-2 Supercomputer.

Construction and Application of the WEB Interface of Blast Package

Construction and application of cDNA library sequence analysis platform based on Linux

BioSeq-Analysis: a platform for DNA, RNA and protein sequence analysis based on machine learning approaches

Development of Laboratory Sequence Analysis Software Based on WWW and UNIX

Construction and application of a high-throughput analysis system for peanut transcriptome

Robust and Rapid Algorithms Facilitate Large-Scale Whole Genome Sequencing Downstream Analysis in an Integrative Framework

ProSeq4: A user‐friendly multiplatform program for preparation and analysis of large‐scale DNA polymorphism datasets

Pse-Analysis: a python package for DNA/RNA and protein/peptide sequence analysis based on pseudo components and kernel methods

The Challenge of Scaling Genome Big Data Analysis Software on TH-2 Supercomputer

Computing Platforms for Big Biological Data Analytics: Perspectives and Challenges

[An Introduction of Several Programs Used in Genomic Analysis].

Massive Genomic Data Processing and Deep Analysis

Build a Bioinformatic Analysis Platform and Apply it to Routine Analysis of Microbial Genomics and Comparative Genomics

Hierarchical DNN with Heterogeneous Computing Enabled High-Performance DNA Sequencing

KSI：a DNA sequence matching library for terabyte scale bio-data

HPTree: Reconstructing Phylogenetic Trees for Ultra-Large Unaligned DNA Sequences Via NJ Model and Hadoop