Abstract:MapReduce, the popular programming paradigm for large-scale data processing, has traditionally been deployed over tightly-coupled clusters where the data is already locally available. The assumption that the data and compute resources are available in a single central location, however, no longer holds for many emerging applications in commercial, scientific and social networking domains, where the data is generated in a geographically distributed manner. Further, the computational resources needed for carrying out the data analysis may be distributed across multiple data centers or community resources such as Grids. In this paper, we develop a modeling framework to capture MapReduce execution in a highly distributed environment comprising distributed data sources and distributed computational resources. This framework is flexible enough to capture several design choices and performance optimizations for MapReduce execution. We propose a model-driven optimization that has two key features: (i) it is end-to-end as opposed to myopic optimizations that may only make locally optimal but globally suboptimal decisions, and (ii) it can control multiple MapReduce phases to achieve low runtime, as opposed to single-phase optimizations that may control only individual phases. Our model results show that our optimization can provide nearly 82% and 64% reduction in execution time over myopic and single-phase optimizations, respectively. We have modified Hadoop to implement our model outputs, and using three different MapReduce applications over an 8-node emulated PlanetLab testbed, we show that our optimized Hadoop execution plan achieves 31-41% reduction in runtime over a vanilla Hadoop execution. Our model-driven optimization also provides several insights into the choice of techniques and execution parameters based on application and platform characteristics.

Distributed data management using MapReduce

Distributed Affinity Propagation Clustering Based on MapReduce

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Query optimization for massively parallel data processing.

A Survey on Geographically Distributed Big-Data Processing using MapReduce

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Massive Image Data Management Using Hbase And Mapreduce

Optimizing MapReduce for Highly Distributed Environments

An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications

An Analytical Performance Model of MapReduce

An Uncoupled Data Process and Transfer Model for MapReduce.

Progressive online aggregation in a distributed stream system

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

Distributed XPath Query Processing over Large XML Data Based on MapReduce Framework

Distributed Log Analysis on the Cloud Using MapReduce

MAP-JOIN-REDUCE: Toward Scalable and Efficient Data Analysis on Large Clusters

Hadoop, MapReduce and HDFS: A Developers Perspective

Hybrid storage architecture and efficient MapReduce processing for unstructured data

Research of a MapReduce Model to Process the Traffic Big Data