Abstract:MapReduce, a parallel computational model, has been widely used in processing big data in a distributed cluster. Consisting of alternate map and reduce phases, MapReduce has to shuffle the intermediate data generated by mappers to reducers. The key challenge of ensuring balanced workload on MapReduce is to reduce partition skew among reducers without detailed distribution information on mapped data. In this paper, we propose an incremental data allocation approach to reduce partition skew among reducers on MapReduce. The proposed approach divides mapped data into many micro-partitions and gradually gathers the statistics on their sizes in the process of mapping. The micropartitions are then incrementally allocated to reducers in multiple rounds. We propose to execute incremental allocation in two steps, micro-partition scheduling and micro-partition allocation. We propose a Markov decision process (MDP) model to optimize the problem of multiple-round micropartition scheduling for allocation commitment. We present an optimal solution with the time complexity of O ( K · N 2 ), in which K represents the number of allocation rounds and N represents the number of micro-partitions. Alternatively, we also present a greedy but more efficient algorithm with the time complexity of O ( K · N ln N ). Then, we propose a minmax programming model to handle the allocation mapping between micro-partitions and reducers, and present an effective heuristic solution due to its NP-completeness. Finally, we have implemented the proposed approach on Hadoop, an open-source MapReduce platform, and empirically evaluated its performance. Our extensive experiments show that compared with the state-of-the-art approaches, the proposed approach achieves considerably better data load balance among reducers as well as overall better parallel performance.

Scheduling Divisible Loads from Multiple Input Sources in MapReduce

Scheduling divisible reduce tasks in MapReduce

Load scheduling for distributed edge computing: A communication-computation tradeoff

Distributed Scheduling Strategies for Processing Multiple Divisible Loads with Unknown Network Resources

An Incremental Partitioning Strategy for Data Balance on MapReduce

Reducing Partition Skew on MapReduce: an Incremental Allocation Approach

Optimization of Computing Time for Sequential MapReduce Jobs.

Communication-Efficient Computation Load Scheduling for Delay-Constrained Services

Reliable Estimation of Execution Time of MapReduce Program

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

Scheduling Framework and Algorithms for Large-Scale Divisible Load Processing with Multi-Dimensional QoS Constraints

Optimal Divisible Load Scheduling for Resource-Sharing Network

Survey on MapReduce Scheduling Algorithms

A Game Theory Based MapReduce Scheduling Algorithm

A novel run-time load balancing method for MapReduce

Optimization of Task Assignment Strategy for Map-Reduce

An Analytical Performance Model of MapReduce

MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers

A distributed scheduling strategy for multiple divisible loads with unknown network resources

On Traffic-Aware Partition and Aggregation in MapReduce for Big Data Applications

Multi-objective Scheduling for Divisible Load in Heterogeneous Distributed System.