Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.

Multimedia Video Information Retrieval Based on MapReduce under Cloud Computing.

Multimedia Information Retrieval System Design Based on Cloud Computing

A Distributed Video Management Cloud Platform Using Hadoop.

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Experiences On A Video Management Cloud Based On Hadoop

Energy-efficient Multi-task Scheduling Based on MapReduce for Cloud Computing

Multimedia Applications and Security in MapReduce: Opportunities and Challenges

Parallelizing video transcoding using Map-Reduce-based cloud computing

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Application of Cloud Video Information Processing Technology in Alleviating the Food Safety Trust Crisis

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Distributed Video Processing Platform Based on Map Reduce

GPU-based MapReduce for large-scale near-duplicate video retrieval

GPU-based MapReduce for Large-Scale Near-Duplicate Video Retrieval.

Cloud computing model for big data processing and performance optimization of multimedia communication

Prediction-Based and Locality-Aware Task Scheduling for Parallelizing Video Transcoding over Heterogeneous MapReduce Cluster

Location-Aware MapReduce in Virtual Cloud

Personalized Overseas Chinese Education Model Based on Map-Reduce Model of Cloud Computing

Research on method for extracting large-scale social network based on Mapreduce