Abstract:Nowadays, the rise of various network applications makes network traffic become increasingly complex, which brings more stringent requirements to traffic engineering (TE). Although the state-of-the-art TE approaches based on deep reinforcement learning (DRL) or traditional methods can generate optimal solutions for fixed traffic matrices, they cannot converge fast enough to provide real-time optimization in real networks either because of excessive computation times or high communication overheads. Moreover, due to the dynamically changing traffic load on the network, it is also challenging to achieve optimization of maximum link utilization (MLU) and end-to-end delay at the same time since these two optimization objectives may be conflicting, especially when the network is under a low traffic load, which makes the modeling very difficult. To meet these challenges, we present RT-TE, a TE system based on DRL and distributed message-passing between intelligent agents that can achieve real-time optimization for both MLU and end-to-end delay. To reduce the communication time due to link propagation delay during the optimization process, we design a proactive message-passing mechanism that allows agents to use partial messages to compute the routing policy while maintaining the optimization performance. Additionally, to achieve the tradeoff between the two optimization objectives, we model the propagation delay into the DRL model and design a multi-objective training framework with parameter transfer for training. Based on theoretical modeling, we can find the best tradeoff between the two objectives. Moreover, to improve the model's generalization for various traffic flows, we use a GNN model to generate the rewards of the DRL model, which greatly speeds up the training phase and allows us to feed massive amounts of data into the model. Through evaluations of real-world network topologies, our approach shows a 10%-20% improvement in optimizing MLU under short traffic-changing intervals and yields a 9%-13% improvement in optimizing end-to-end delay compared to state-of-the-art approaches.

An accelerated end-to-end method for solving routing problems

A Deep Reinforcement Learning Based Real-Time Solution Policy for the Traveling Salesman Problem

Step-Wise Deep Learning Models for Solving Routing Problems.

An Approach to Combine the Power of Deep Reinforcement Learning with a Graph Neural Network for Routing Optimization

Solving Optimization Problems Through Fully Convolutional Networks: an Application to the Traveling Salesman Problem

Reinforcement Learning for Solving Colored Traveling Salesman Problems: an Entropy-Insensitive Attention Approach

Learning Improvement Heuristics for Solving Routing Problems

GROM: A Generalized Routing Optimization Method with Graph Neural Network and Deep Reinforcement Learning

Learning to Solve Capacitated Arc Routing Problems by Policy Gradient

An Efficient Learning-based Solver Comparable to Metaheuristics for the Capacitated Arc Routing Problem

A Columnar Competitive Model with Simulated Annealing for Solving Combinatorial Optimization Problems

A Deep Reinforcement Learning Approach for Global Routing

G-Routing: Graph Neural Networks-Based Flexible Online Routing

GASE: Graph Attention Sampling with Edges Fusion for Solving Vehicle Routing Problems

Online Vehicle Routing With Neural Combinatorial Optimization and Deep Reinforcement Learning

GLOP: Learning Global Partition and Local Construction for Solving Large-scale Routing Problems in Real-time

Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Opt

Fast Policy Convergence for Traffic Engineering with Proactive Distributed Message-Passing

Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems

A Graph Convolutional Encoder and Multi-Head Attention Decoder Network for TSP Via Reinforcement Learning

CLSDRL:A Routing Optimization Method for Traffic Feature Extraction