Abstract:The past few years have witnessed a surge of interest towards deep reinforcement learning (DRL) in computer networks. With extraordinary ability of feature extraction, DRL has the potential to re-engineer the fundamental resource allocation problems in networking without relying on pre-programmed models or assumptions about dynamic environments. However, such black-box systems suffer from poor robustness, showing high performance variance and poor tail performance. In this work, we propose a unified Teacher-Student learning framework that harnesses rich domain knowledge to improve robustness. The domain-specific algorithms, less performant but more trustable than DRL, play the role of teachers providing advice at critical states; the student neural network is steered to maximize the expected reward as usual and mimic the teacher's advice meanwhile. The Teacher-Student method comprises of three modules where the confidence check module locates wrong decisions and risky decisions, the reward shaping module designs a new updating function to stimulate the learning of student network, and the prioritized experience replay module to effectively utilize the advised actions. We further implement our Teacher-Student framework in existing video streaming (Pensieve), load balancing (DeepLB), and TCP congestion control (Aurora). Experimental results manifest that the proposed approach reduces the performance standard deviation of DeepLB by 37%; it improves the 90th, 95th, and 99th tail performance of Pensieve by 7.6%, 8.8%, and 10.7% respectively; and it accelerates the growth rate of Aurora by 2x at the initial stage, and achieves a more stable performance in dynamic environments.

Summary and Discussions

Learning Network Decisions

When Machine Learning Meets Congestion Control: A Survey and Comparison

A comprehensive survey on machine learning for networking: evolution, applications and research opportunities

Reinforcement Learning based Interconnection Routing for Adaptive Traffic Optimization

Learning to Branch: Accelerating Resource Allocation in Wireless Networks

Learning from Networks

Machine Learning With Computer Networks: Techniques, Datasets, and Models

Machine Learning for Networking: Workflow, Advances and Opportunities

An Introduction to Advanced Machine Learning : Meta Learning Algorithms, Applications and Promises

Evolutionary Learning: Advances in Theories and Algorithms

Toward Intelligent Network Optimization in Wireless Networking: An Auto-Learning Framework

Learning for Decision and Control in Stochastic Networks

The Power of Online Learning in Stochastic Network Optimization.

Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective

Review of Learning-Assisted Power System Optimization

When Meta-Learning Meets Online and Continual Learning: A Survey

Enabling Robust DRL-Driven Networking Systems Via Teacher-Student Learning

The Confluence of Networks, Games and Learning

Optimization for deep learning: theory and algorithms

Learning Sub-Second Routing Optimization in Computer Networks requires Packet-Level Dynamics