Abstract:MapReduce, the popular programming paradigm for large-scale data processing, has traditionally been deployed over tightly-coupled clusters where the data is already locally available. The assumption that the data and compute resources are available in a single central location, however, no longer holds for many emerging applications in commercial, scientific and social networking domains, where the data is generated in a geographically distributed manner. Further, the computational resources needed for carrying out the data analysis may be distributed across multiple data centers or community resources such as Grids. In this paper, we develop a modeling framework to capture MapReduce execution in a highly distributed environment comprising distributed data sources and distributed computational resources. This framework is flexible enough to capture several design choices and performance optimizations for MapReduce execution. We propose a model-driven optimization that has two key features: (i) it is end-to-end as opposed to myopic optimizations that may only make locally optimal but globally suboptimal decisions, and (ii) it can control multiple MapReduce phases to achieve low runtime, as opposed to single-phase optimizations that may control only individual phases. Our model results show that our optimization can provide nearly 82% and 64% reduction in execution time over myopic and single-phase optimizations, respectively. We have modified Hadoop to implement our model outputs, and using three different MapReduce applications over an 8-node emulated PlanetLab testbed, we show that our optimized Hadoop execution plan achieves 31-41% reduction in runtime over a vanilla Hadoop execution. Our model-driven optimization also provides several insights into the choice of techniques and execution parameters based on application and platform characteristics.

Optimization for Iterative Queries on MapReduce

Join Query Optimization Based on MapReduce under Skewed Data

Query optimization for massively parallel data processing.

AQUA+: Query Optimization for Hybrid Database-MapReduce System.

Query grouping-based multi-query optimization framework for interactive SQL query engines on Hadoop.

Optimizing MapReduce for Highly Distributed Environments

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Iterative Query Processing based on Unified Optimization Techniques

Quadratic Interpolation Optimization (QIO): A new optimization algorithm based on generalized quadratic interpolation and its applications to real-world engineering problems

Optimization and reconstruction shuffle in MapReduce

Exploiting Shared Sub-Expression and Materialized View Reuse for Multi-Query Optimization

MapReduce Job Optimization: A Mapping Study

AQP++: Connecting Approximate Query Processing with Aggregate Precomputation for Interactive Analytics

New Distributed Spatial Query Optimization Approach by Using Query Analyzer

Efficient Query Re-optimization with Judicious Subquery Selections

Quantum-Informed Recursive Optimization Algorithms

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

Submodular Optimization in the MapReduce Model

Performance Optimization for Short MapReduce Job Execution in Hadoop

OPTAS: Optimal Data Placement in MapReduce.

Cost-Based Optimization Of Logical Partitions For A Query Workload In A Hadoop Data Warehouse