Abstract:Stochastic Network Optimization (SNO) concerns scheduling in stochastic queueing systems. It has been widely studied in network theory. Classical SNO algorithms require network conditions to be stationary with time, which fails to capture the non-stationary components in many real-world scenarios. Many existing algorithms also assume knowledge of network conditions before decision, which rules out applications where unpredictability presents. Motivated by these issues, we consider Adversarial Network Optimization (ANO) under bandit feedback. Specifically, we consider the task of *i)* maximizing some unknown and time-varying utility function associated to scheduler's actions, where *ii)* the underlying network is a non-stationary multi-hop one whose conditions change arbitrarily with time, and *iii)* only bandit feedback (effect of actually deployed actions) is revealed after decisions. Our proposed `UMO2` algorithm ensures network stability and also matches the utility maximization performance of any "mildly varying" reference policy up to a polynomially decaying gap. To our knowledge, no previous ANO algorithm handled multi-hop networks or achieved utility guarantees under bandit feedback, whereas ours can do both. Technically, our method builds upon a novel integration of online learning into Lyapunov analyses: To handle complex inter-dependencies among queues in multi-hop networks, we propose meticulous techniques to balance online learning and Lyapunov arguments. To tackle the learning obstacles due to potentially unbounded queue sizes, we design a new online linear optimization algorithm that automatically adapts to loss magnitudes. To maximize utility, we propose a bandit convex optimization algorithm with novel queue-dependent learning rate scheduling that suites drastically varying queue lengths. Our new insights in online learning can be of independent interest.

When Lyapunov Drift Based Queue Scheduling Meets Adversarial Bandit Learning

Queue Scheduling with Adversarial Bandit Learning

Learning a Discrete Set of Optimal Allocation Rules in a Queueing System with Unknown Service Rate

Adversarial Network Optimization under Bandit Feedback: Maximizing Utility in Non-Stationary Multi-Hop Networks

Learning to Schedule in Non-Stationary Wireless Networks With Unknown Statistics

Scheduling Servers with Stochastic Bilinear Rewards

Open Bandit Processes and Optimal Scheduling of Queueing Networks

Learning-based Scheduling for Information Gathering with QoS Constraints

Decentralized Scheduling with QoS Constraints: Achieving O(1) QoS Regret of Multi-Player Bandits

Better than the Best: Gradient-based Improper Reinforcement Learning for Network Scheduling

Opportunistic Scheduling as Restless Bandits

Scheduling with Rate Adaptation under Incomplete Knowledge of Channel/Estimator Statistics

Delay-Optimal User Scheduling and Inter-Cell Interference Management in Cellular Network via Distributive Stochastic Learning

Learning-based Optimal Quantum Switch Scheduling

Lyapunov-guided Multi-Agent Reinforcement Learning for Delay-Sensitive Wireless Scheduling

Learning to Schedule Online Tasks with Bandit Feedback

Learning While Scheduling in Multi-Server Systems with Unknown Statistics: MaxWeight with Discounted UCB

Jamsa: A Utility Optimal Contextual Online Learning Framework for Anti-Jamming Wireless Scheduling under Reactive Jamming Attack

Queueing Matching Bandits with Preference Feedback

Online Distribution Network Scheduling via Provably Robust Learning Approach

Distributed Flow Scheduling in an Unknown Environment