Abstract:With the development of next-generation sequencing (NGS), DNA/RNA sequencing has become cheaper and more efficient. Today, a whole human genome can be sequenced under $1,000, providing opportunities for large-scale bioinformatic analysis on big datasets. However, most of existing bioinformatic analysis tools are programmed for single server based computing platform and not suitable to process such big datasets. As Hadoop MapReduce and Spark are gaining popularity as cluster computing based big data processing platform, more and more bioinformatic applications start to explore cluster computing platform for large scale data analysis. In this paper we present an in-depth experimental study on deploying Spark clusters for high performance bioinformatic short sequence reconstruction. Our experimental results enable us to answer a number of challenging and yet most frequently asked questions regarding efficient management of bioinformatic data analysis services on Spark systems. Example questions include how to best split big dataset into multiple partitions, and how to distribute data partitions and bioinformatic analysis tasks on a Spark cluster for carrying out a high performance distributed analysis job? What types of memory models are effective for bioinformatic data analysis services on a Spark cluster? Why do different bioinformatic data analysis operations exhibit different throughput performance on the same Spark cluster? We conjecture that this experimental study not only demonstrates the feasibility of high performance bioinformatic data analysis on Spark platform, but also will help bioinformatic application developers to make more informed decisions on both design and configuration of Spark Cluster, managing and tuning parameters of Spark runtime system for enhancing the performance of large scale big data analytics.

Research on Query Analysis and Optimization Based on Spark

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

Distributed High-Dimension Matrix Operation Optimization on Spark

Optimization of Spark Storage Solutions

Query Execution Optimization in Spark SQL

Opti Zation Of Spark S O Age Solutions

Query optimization Approach with Shuffle Intermediate Cache Layer for Spark SQL

Query Optimization for Massive RDF Data Based on Spark

Query optimization for massively parallel data processing.

OPTIMIZATION FOR SPARK MISSION PERFORMANCE BASED ON DATA CHARACTERISTICS

Distributed data analysis and processing platform based on Pig_Spark

SrSpark: Skew-resilient spark based on adaptive parallel processing

A Survey on Spark Ecosystem for Big Data Processing

Research on Temporal Query Expansion and Temporal Index Optimization Based on Spark

A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications

An Experimental Study of a Biosequence Big Data Analysis Service

Workload Driven Comparison And Optimization Of Hive And Spark Sql

SparkRDF: Elastic Discreted RDF Graph Processing Engine with Distributed Memory

A New Scheme for Cache Optimization Based on Cluster Computing Framework Spark

Indexing for Large Scale Data Querying Based on Spark SQL

Comparative Study on MapReduce and Spark for Big Data Analytics