Abstract:Due to the intensive computation requirements of emerging applications and the limited computational capability of edge computing servers, the computation task must be executed on multiple edge servers in a distributive and cooperative manner. However, the large amount of information exchanged among the edge servers is a major obstacle for improving the computing performance. By utilizing the excess computational resource, coded MapReduce provides an effective approach to reduce the communication load. In this paper, we develop a stochastic load scheduling framework to complete the computation tasks with coded MapReduce considering the intrinsic tradeoff between the communication and computation loads. Our goal is to minimize the communication load under time-varying excess computational resources. We first reduce this problem to a task scheduling problem by exploiting the property of the computing repetition in the coded MapReduce framework. Since the task scheduling problem is still a stochastic optimization problem, it is generally difficult to solve. In the offline setting, we obtain the optimal computation load scheduling algorithm by adopting the augmented Lagrangian method. In the online setting, we derive a worst-case performance bound of the online equal task scheduling (ETS) algorithm by using competitive analysis. Furthermore, we make full use of past state information of computing resources for pre-planing and propose an improved algorithm based on the ETS algorithm in a learning manner. Finally, our proposed algorithm is evaluated by simulation to demonstrate that the proposed algorithms are superior over the conventional algorithms, and the performance gap between the online and offline algorithms is fairly small.

A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters.

Data Based Application Partitioning and Workload Balance in Distributed Environment

Load Balancing In Heterogeneous Mapreduce Environments

Distributed Model Based on Data Partition and Load Balance Algorithm

Fuzzy Clustering with Feature Weight Preferences for Load Balancing in Cloud

Dynamic Load Balance For Hot-Spot And Unbalance Region Problems In Hbase

A load balance algorithm based on nodes performance in Hadoop cluster

A Heterogeneity-aware Data Distribution and Rebalance Method in Hadoop Cluster

Combined Load Balancing and Energy Efficiency in Hadoop

Optimization and Realization of Hadoop Load Balance Mechanism Based on a Multivariable Dynamic Evaluate Threshold Algorithm

Dynamic Load-Balancing and High Performance Communication in Jcluster

Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters

Load scheduling for distributed edge computing: A communication-computation tradeoff

A Dynamic Client-Side Load Balancing Mechanism

OFScheduler: A Dynamic Network Optimizer for MapReduce in Heterogeneous Cluster

DALB: A Dynamic Application-Sensitive Load Balancing Algorithm

Performance of Dynamic Load Balancing Algorithm on Cluster of Workstations and PCs

Moving Hadoop into the Cloud with Flexible Slot Management and Speculative Execution

Load Balancing Algorithms for Big Data Flow Classification Based on Heterogeneous Computing in Software Definition Networks

An Effective Dynamic Load Balancing Method

An Adaptive Approach to Better Load Balancing in a Consumer-Centric Cloud Environment