Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.

SemanMR: big data processing framework based on semantics

A Semantic++ MapReduce: A Preliminary Report

A Semantic++ MapReduce Parallel Programming Model.

GENMR: Generalized Query Processing through Map Reduce In Cloud Database Management System

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

MR-ELM: a MapReduce-based framework for large-scale ELM training in big data era

Research on Semantic++ Computing Based on Big Data Environment

Distributed data management using MapReduce

MtMR: Ensuring MapReduce Computation Integrity with Merkle Tree-Based Verifications

Semantic-based Intelligent Data Clean Framework for Big Data

Semantic-based Big Data integration framework using scalable distributed ontology matching strategy

An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications

Semantic HMC for Big Data Analysis

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Distributed Semantic Web Data Management in HBase and MySQL Cluster

The performance of MapReduce: an in-depth study

MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

ConMR: Concurrent MapReduce Programming Model for Large Scale Shared-Data Applications

An Approach in Big Data Analytics to Improve the Velocity of Unstructured Data Using MapReduce

A Survey of Semantics-Aware Performance Optimization for Data-Intensive Computing