Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.

LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

Distributed Affinity Propagation Clustering Based on MapReduce

Community Mining From Multi-Relational Networks

Exploring Computation Locality of Graph Mining Algorithms on MapReduce

Community structure mining in big data social media networks with MapReduce

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A Parallel Computing Model for Large-Graph Mining with MapReduce.

DataMPI: Extending MPI to Hadoop-Like Big Data Computing

Large-Scale Social Network Analysis Based on MapReduce

Large scale microblog mining using distributed MB-LDA.

MR-ELM: a MapReduce-based framework for large-scale ELM training in big data era

Combination of in-memory graph computation with mapreduce: a subgraph-centric method of pagerank

LLMapReduce: Multi-Level Map-Reduce for High Performance Data Analysis

The performance of MapReduce: an in-depth study

Detecting Overlapping Community Structures In Networks With Global Partition And Local Expansion

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Detecting Associations in Large Dataset on MapReduce

An Analytical Performance Model of MapReduce

Scalable Community Discovery of Large Networks

Research on method for extracting large-scale social network based on Mapreduce