Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.

Research on method for extracting large-scale social network based on Mapreduce

Large-Scale Social Network Analysis Based on MapReduce

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A Parallel Computing Model for Large-Graph Mining with MapReduce.

Parallel Approach and Platform for Large-Scale WEB Data Extraction

Community structure mining in big data social media networks with MapReduce

X-RIME: Cloud-Based Large Scale Social Network Analysis

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

Exploratory Analysis for Big Social Data Using Deep Network

LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

RESEARCH ON GIS MASSIVE TRAFFIC DATA ANALYSIS PLATFORM BASED ON HADOOP

The performance of MapReduce: an in-depth study

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Xml Structural Similarity Search Using Mapreduce

Research of urban traffic carbon emission data mining based on Hadoop

Network Public Opinion Monitoring Based on MapReduce

Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

Distributed Centrality Analysis of Social Network Data Using MapReduce

Parallelization in Extracting Fresh Information from Online Social Network

CloudCD:Cloud-Based Community Discovery of Social Network