Abstract:The past few years have witnessed a surge of interest towards deep reinforcement learning (DRL) in computer networks. With extraordinary ability of feature extraction, DRL has the potential to re-engineer the fundamental resource allocation problems in networking without relying on pre-programmed models or assumptions about dynamic environments. However, such black-box systems suffer from poor robustness, showing high performance variance and poor tail performance. In this work, we propose a unified Teacher-Student learning framework that harnesses rich domain knowledge to improve robustness. The domain-specific algorithms, less performant but more trustable than DRL, play the role of teachers providing advice at critical states; the student neural network is steered to maximize the expected reward as usual and mimic the teacher's advice meanwhile. The Teacher-Student method comprises of three modules where the confidence check module locates wrong decisions and risky decisions, the reward shaping module designs a new updating function to stimulate the learning of student network, and the prioritized experience replay module to effectively utilize the advised actions. We further implement our Teacher-Student framework in existing video streaming (Pensieve), load balancing (DeepLB), and TCP congestion control (Aurora). Experimental results manifest that the proposed approach reduces the performance standard deviation of DeepLB by 37%; it improves the 90th, 95th, and 99th tail performance of Pensieve by 7.6%, 8.8%, and 10.7% respectively; and it accelerates the growth rate of Aurora by 2x at the initial stage, and achieves a more stable performance in dynamic environments.

Learning by Reusing Previous Advice: a Memory-Based Teacher–student Framework

Learning by Reusing Previous Advice in Teacher-Student Paradigm.

Teacher-Student Framework: a Reinforcement Learning Approach

Learning on a Budget via Teacher Imitation

Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework

Methodical Advice Collection and Reuse in Deep Reinforcement Learning

Action Advising with Advice Imitation in Deep Reinforcement Learning

A federated advisory teacher–student framework with simultaneous learning agents

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Self-Attention Guided Advice Distillation in Multi-Agent Deep Reinforcement Learning

Teachable Reinforcement Learning via Advice Distillation

Learning to Teach Reinforcement Learning Agents

A Deep Reinforcement Learning Framework for Instructional Sequencing

A Q-values Sharing Framework for Multiagent Reinforcement Learning under Budget Constraint

Improving Policy Generalization for Teacher-Student Reinforcement Learning.

Teacher Agent: A Knowledge Distillation-Free Framework for Rehearsal-based Video Incremental Learning

A Q-values Sharing Framework for Multiple Independent Q-learners

Enabling Robust DRL-Driven Networking Systems Via Teacher-Student Learning

DRL-SRS: A Deep Reinforcement Learning Approach for Optimizing Spaced Repetition Scheduling

Integrating human learning and reinforcement learning: A novel approach to agent training