Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.

Network Public Opinion Monitoring Based on MapReduce

Distributed Affinity Propagation Clustering Based on MapReduce

Construct High Performance Text Clustering Systems Based on Map-Reduce

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Using Visualization to Improve Clustering Analysis on Heterogeneous Information Network.

Large-Scale Social Network Analysis Based on MapReduce

A Public Opinion Monitoring System Based on Big Data Technology

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Community structure mining in big data social media networks with MapReduce

Research on the Dynamic Monitoring System Model of University Network Public Opinion under the Big Data Environment

Research on the Computer Informatization in Multimedia Public Opinion Monitoring

An efficient PAM spatial clustering algorithm based on MapReduce

A 2-Tier Clustering Algorithm with Map-Reduce

Distributed structural clustering on large graph

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

Parallel Link Prediction in Complex Network Using MapReduce

5New density clustering algorithm based on MapReduce

A Parallel Computing Model for Large-Graph Mining with MapReduce.

Distributed Centrality Analysis of Social Network Data Using MapReduce

The performance of MapReduce: an in-depth study

Distributed data management using MapReduce