Abstract:The most recent development of the Internet of Things brings massive timely sensitive and bursty data flows. Also, joint optimization on storage, computation, and communication is in need for multiaccess edge computing frameworks. The adaptive network control has been explored using deep reinforcement learning (RL), but it is not sufficient for bursty network traffic flows, especially when the network traffic pattern may change over time. We formulate the routing control in an environment with time-variant link delays as a Lyapunov optimization problem. We identify that there is a tradeoff between optimization performance and modeling accuracy when the propagation delays are included. We propose a novel deep RL (DRL)-based adaptive network routing method to tackle the issues mentioned above. A Lyapunov optimization technique is used to reduce the upper bound of the Lyapunov drift, improving queuing stability in networked systems. By modeling the network traffic pattern using the Markovian arrival process, we show that network routing problems can be modeled as Markov decision processes and value-iteration-based RL methods can be used to solve them. We design a blockchain-based protocol using proof of elapsed time consensus mechanism to ensure a trustworthy network statistics information exchange for the routing framework. Experiment results show that the proposed method can learn a routing policy and adapt to the changing environment. The proposed method outperforms the baseline backpressure method in multiple settings and converges faster than existing methods. Moreover, the DRL module can effectively learn a better estimation of the long-term Lyapunov drift and penalty functions, providing superior results in terms of the backlog size, end-to-end latency, age of information, and throughput. Furthermore, the blockchain-based network statistics exchange can provide the routing framework against malicious nodes. In addition, the proposed model performs well under various topologies, and thus can be used in general cases.

Deep Distributional Reinforcement Learning-Based Adaptive Routing with Guaranteed Delay Bounds

Deep Reinforcement Learning Based Dynamic Routing Optimization for Delay-Sensitive Applications

Deep Reinforcement Learning for On-demand Intelligent Routing in Deterministic Networks

Deep Reinforcement Learning for Router Selection in Network with Heavy Traffic

Reinforcement Learning Empowered QoS-aware Adaptive Q-Routing in Ad-hoc Networks.

QoS Routing Optimization Based on Deep Reinforcement Learning in SDN

Adaptive and Robust Network Routing Based on Deep Reinforcement Learning with Lyapunov Optimization.

Deep Reinforcement Learning-Based Deterministic Routing and Scheduling for Mixed-Criticality Flows

Adaptive and Robust Routing with Lyapunov-Based Deep RL in MEC Networks Enabled by Blockchains.

DRLAR: A Deep Reinforcement Learning-Based Adaptive Routing Framework for Network-on-chips

A Deep Reinforcement Learning-based Routing Scheme with Two Modes for Dynamic Networks

Multi-Agent DRL-Based Deadline-Aware Routing for Deterministic Wide-Area Networks

SDDRL-SR: A High-Reliability Satellite Routing Algorithm Based on Deep Reinforcement Learning

Deep Learning Empowered QoS-aware Adaptive Routing Algorithm in Wireless Networks

Toward Packet Routing With Fully Distributed Multiagent Deep Reinforcement Learning

Energy-efficient and delay-guaranteed routing algorithm for software-defined wireless sensor networks: A cooperative deep reinforcement learning approach

Deep Reinforcement Learning-based Dynamic Bandwidth Allocation in Weighted Fair Queues of Routers.

An Intelligent SDWN Routing Algorithm Based on Network Situational Awareness and Deep Reinforcement Learning

Toward Packet Routing with Fully-distributed Multi-agent Deep Reinforcement Learning

A Hybrid Deep Reinforcement Learning Routing Method under Dynamic and Complex Traffic with Software Defined Networking

DRL-based routing algorithm with guaranteed loss, latency and bandwidth in SDN networks: Application of online video conferencing