Abstract:We present an overview of scalable load balancing algorithms which provide favorable delay performance in large-scale systems, and yet only require minimal implementation overhead. Aimed at a broad audience, the paper starts with an introduction to the basic load balancing scenario, consisting of a single dispatcher where tasks arrive that must immediately be forwarded to one of $N$ single-server queues. A popular class of load balancing algorithms are so-called power-of-$d$ or JSQ($d$) policies, where an incoming task is assigned to a server with the shortest queue among $d$ servers selected uniformly at random. This class includes the Join-the-Shortest-Queue (JSQ) policy as a special case ($d = N$), which has strong stochastic optimality properties and yields a mean waiting time that vanishes as $N$ grows large for any fixed subcritical load. However, a nominal implementation of the JSQ policy involves a prohibitive communication burden in large-scale deployments. In contrast, a random assignment policy ($d = 1$) does not entail any communication overhead, but the mean waiting time remains constant as $N$ grows large for any fixed positive load. In order to examine the fundamental trade-off between performance and implementation overhead, we consider an asymptotic regime where $d(N)$ depends on $N$. We investigate what growth rate of $d(N)$ is required to match the performance of the JSQ policy on fluid and diffusion scale. The results demonstrate that the asymptotics for the JSQ($d(N)$) policy are insensitive to the exact growth rate of $d(N)$, as long as the latter is sufficiently fast, implying that the optimality of the JSQ policy can asymptotically be preserved while dramatically reducing the communication overhead. We additionally show how the communication overhead can be reduced yet further by the so-called Join-the-Idle-Queue scheme, leveraging memory at the dispatcher.

On Choosing a Task Assignment Policy for a Distributed Server System

Task assignment policies in distributed server systems: A survey

SDSSE: A Self-Driven RPC Load Balancing Method in Datacenter

Optimal Rate-Matrix Pruning For Large-Scale Heterogeneous Systems

Optimal Scheduling in the Multiserver-job Model under Heavy Traffic

Evaluating load balancing policies for performance and energy-efficiency

Scalable Load Balancing in Networked Systems: Universality Properties and Stochastic Coupling Methods

Dynamic Server Allocation over Time Varying Channels with Switchover Delay

A Dynamic Task Assignment Approach Based on Individual Worklists for Minimizing the Cycle Time of Business Processes

Switching Scheduling Policies and Their Balanceabililty

Theoretically Guaranteed Online Workload Dispatching for Deadline-Aware Multi-Server Jobs

Task Assignment in Loosely‐coupled Multiprocessor Systems

Utility maximizing load balancing policies

Balanced Splitting: A Framework for Achieving Zero-wait in the Multiserver-job Model

Kinetically Stable Task Assignment for Networks of Microservers

Effectiveness of distributed stateless network server selection under strict latency constraints

Large-scale multi-robot task allocation via dynamic partitioning and distribution

Generalized Parallel-Server Fork-Join Queues with Dynamic Task Scheduling.

Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers

A Restless Bandit Model for Energy-Efficient Job Assignments in Server Farms

The Balanceabililty Of Amm Scheduling Policy For Gss Ystems