Abstract:As a widely-used parallel computing framework for big data processing today, the Hadoop MapReduce framework puts more emphasis on high-throughput of data than on low-latency of job execution. However, today more and more big data applications developed with MapReduce require quick response time. As a result, improving the performance of MapReduce jobs, especially for short jobs, is of great significance in practice and has attracted more and more attentions from both academia and industry. A lot of efforts have been made to improve the performance of Hadoop from job scheduling or job parameter optimization level. In this paper, we explore an approach to improve the performance of the Hadoop MapReduce framework by optimizing the job and task execution mechanism. First of all, by analyzing the job and task execution mechanism in MapReduce framework we reveal two critical limitations to job execution performance. Then we propose two major optimizations to the MapReduce job and task execution mechanisms: first, we optimize the setup and cleanup tasks of a MapReduce job to reduce the time cost during the initialization and termination stages of the job; second, instead of adopting the loose heartbeat-based communication mechanism to transmit all messages between the JobTracker and TaskTrackers, we introduce an instant messaging communication mechanism for accelerating performance-sensitive task scheduling and execution. Finally, we implement SHadoop, an optimized and fully compatible version of Hadoop that aims at shortening the execution time cost of MapReduce jobs, especially for short jobs. Experimental results show that compared to the standard Hadoop, SHadoop can achieve stable performance improvement by around 25% on average for comprehensive benchmarks without losing scalability and speedup. Our optimization work has passed a production-level test in Intel and has been integrated into the Intel Distributed Hadoop (IDH). To the best of our knowledge, this work is the first effort that explores on optimizing the execution mechanism inside map/reduce tasks of a job. The advantage is that it can complement job scheduling optimizations to further improve the job execution performance.

An Efficient MapReduce Framework for Intel MIC Cluster.

Optimizing the MapReduce Framework for CPU-MIC Heterogeneous Cluster

Micmr: an Efficient Mapreduce Framework for Cpu-Mic Heterogeneous Architecture

Optimizing the MapReduce Framework on Intel Xeon Phi Coprocessor

Tiled-MapReduce: Efficient and Flexible MapReduce Processing on Multicore with Tiling

Tiled-MapReduce

Accelerating MapReduce on Commodity Clusters: an SSD-Empowered Approach

An Approach to Enhance the Performance of Large‐scale Structural Analysis on CPU‐MIC Heterogeneous Clusters

Tiled-Mapreduce: Optimizing Resource Usages Of Data-Parallel Applications On Multicore With Tiling

A MapReduce Computing Framework Based on GPU Cluster.

The performance of MapReduce: an in-depth study

Cluster-level tuning of a shallow water equation solver on the Intel MIC architecture

An Analytical Performance Model of MapReduce

A-MapCG: an Adaptive MapReduce Framework for GPUs.

NO2: Speeding Up Parallel Processing of Massive Compute-Intensive Tasks

FT-Offload: A Scalable Fault-Tolerance Programing Model on MIC Cluster

The Performance of MapReduce

Thread Mapping and Parallel Optimization for MIC Heterogeneous Parallel Systems.

A CPU/MIC Collaborated Parallel Framework for GROMACS on Tianhe-2 Supercomputer

MIC-THPCM: MIC-Based Heterogeneous Parallel Optimization for Axial Compressor Rotor

SHadoop: Improving MapReduce Performance by Optimizing Job Execution Mechanism in Hadoop Clusters