Abstract:Stochastic Network Optimization (SNO) concerns scheduling in stochastic queueing systems. It has been widely studied in network theory. Classical SNO algorithms require network conditions to be stationary with time, which fails to capture the non-stationary components in many real-world scenarios. Many existing algorithms also assume knowledge of network conditions before decision, which rules out applications where unpredictability presents. Motivated by these issues, we consider Adversarial Network Optimization (ANO) under bandit feedback. Specifically, we consider the task of *i)* maximizing some unknown and time-varying utility function associated to scheduler's actions, where *ii)* the underlying network is a non-stationary multi-hop one whose conditions change arbitrarily with time, and *iii)* only bandit feedback (effect of actually deployed actions) is revealed after decisions. Our proposed `UMO2` algorithm ensures network stability and also matches the utility maximization performance of any "mildly varying" reference policy up to a polynomially decaying gap. To our knowledge, no previous ANO algorithm handled multi-hop networks or achieved utility guarantees under bandit feedback, whereas ours can do both. Technically, our method builds upon a novel integration of online learning into Lyapunov analyses: To handle complex inter-dependencies among queues in multi-hop networks, we propose meticulous techniques to balance online learning and Lyapunov arguments. To tackle the learning obstacles due to potentially unbounded queue sizes, we design a new online linear optimization algorithm that automatically adapts to loss magnitudes. To maximize utility, we propose a bandit convex optimization algorithm with novel queue-dependent learning rate scheduling that suites drastically varying queue lengths. Our new insights in online learning can be of independent interest.

Queue Scheduling with Adversarial Bandit Learning

When Lyapunov Drift Based Queue Scheduling Meets Adversarial Bandit Learning

Logarithmically Adaptive Queue Scheduling Algorithm

Learning a Discrete Set of Optimal Allocation Rules in a Queueing System with Unknown Service Rate

Queueing Matching Bandits with Preference Feedback

Decentralized Scheduling with QoS Constraints: Achieving O(1) QoS Regret of Multi-Player Bandits

Adversarial Network Optimization under Bandit Feedback: Maximizing Utility in Non-Stationary Multi-Hop Networks

Open Bandit Processes and Optimal Scheduling of Queueing Networks

Scheduling Servers with Stochastic Bilinear Rewards

Learning to Schedule in Non-Stationary Wireless Networks With Unknown Statistics

Learning-based Scheduling for Information Gathering with QoS Constraints

Learning While Scheduling in Multi-Server Systems with Unknown Statistics: MaxWeight with Discounted UCB

Scheduling with Rate Adaptation under Incomplete Knowledge of Channel/Estimator Statistics

Delay-Optimal User Scheduling and Inter-Cell Interference Management in Cellular Network via Distributive Stochastic Learning

Learning Optimal Admission Control in Partially Observable Queueing Networks

Quantifying the Cost of Learning in Queueing Systems

Better than the Best: Gradient-based Improper Reinforcement Learning for Network Scheduling

Multi-Agent Multi-Armed Bandit Learning for Online Management of Edge-Assisted Computing

Learning to Schedule Online Tasks with Bandit Feedback

Learning Optimal Scheduling Policy for Remote State Estimation Under Uncertain Channel Condition

Online Offloading in Dense Wireless Networks: an Adversary Multi-armed Bandit Approach