Abstract:Due to the intensive computation requirements of emerging applications and the limited computational capability of edge computing servers, the computation task must be executed on multiple edge servers in a distributive and cooperative manner. However, the large amount of information exchanged among the edge servers is a major obstacle for improving the computing performance. By utilizing the excess computational resource, coded MapReduce provides an effective approach to reduce the communication load. In this paper, we develop a stochastic load scheduling framework to complete the computation tasks with coded MapReduce considering the intrinsic tradeoff between the communication and computation loads. Our goal is to minimize the communication load under time-varying excess computational resources. We first reduce this problem to a task scheduling problem by exploiting the property of the computing repetition in the coded MapReduce framework. Since the task scheduling problem is still a stochastic optimization problem, it is generally difficult to solve. In the offline setting, we obtain the optimal computation load scheduling algorithm by adopting the augmented Lagrangian method. In the online setting, we derive a worst-case performance bound of the online equal task scheduling (ETS) algorithm by using competitive analysis. Furthermore, we make full use of past state information of computing resources for pre-planing and propose an improved algorithm based on the ETS algorithm in a learning manner. Finally, our proposed algorithm is evaluated by simulation to demonstrate that the proposed algorithms are superior over the conventional algorithms, and the performance gap between the online and offline algorithms is fairly small.

MapReduce Scheduler: A 360-degree view

MapReduce scheduling algorithms in Hadoop: a systematic study

Survey on MapReduce Scheduling Algorithms

MapReduce Parallel Programming Model: A State-of-the-Art Survey

Real-Time Scheduling in MapReduce Clusters

A Survey on MapReduce Optimization Technologies

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

A Game Theory Based MapReduce Scheduling Algorithm

Data-Replicas Scheduler for Heterogeneous MapReduce Cluster.

An Efficient Job Scheduling for Map Reduce Clusters

Matchmaking: A New MapReduce Scheduling Technique

Load scheduling for distributed edge computing: A communication-computation tradeoff

Distributed data management using MapReduce

An Adaptive Scheduling Algorithm for Heterogeneous Hadoop Systems

The performance of MapReduce: an in-depth study

Scheduling Divisible Loads from Multiple Input Sources in MapReduce

K%-Fair Scheduling: A Flexible Task Scheduling Strategy for Balancing Fairness and Efficiency in MapReduce Systems

An Analytical Review and Performance Measures of State-of-Art Scheduling Algorithms in Heterogenous Computing Enviornment

Design and Implementation of a Real-time Scheduling Algorithm for MapReduce

MapReduce:a New Programming Model for Distributed Parallel Computing

MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers