Abstract:Topology impacts important network performance metrics, including link utilization, throughput and latency, and is of central importance to network operators. However, due to the combinatorial nature of network topology, it is extremely difficult to obtain an optimal solution, especially since topology planning in networks also often comes with management-specific constraints. As a result, local optimization with hand-tuned heuristic methods from human experts are often adopted in practice. Yet, heuristic methods cannot cover the global topology design space while taking into account constraints, and cannot guarantee to find good solutions. In this paper, we propose a novel deep reinforcement learning (DRL) algorithm, called Advantage Actor Critic-Graph Searching (A2C-GS), for network topology optimization. A2C-GS consists of three novel components, including a verifier to validate the correctness of a generated network topology, a graph neural network (GNN) to efficiently approximate topology rating, and a DRL actor layer to conduct a topology search. A2C-GS can efficiently search over large topology space and output topology with satisfying performance. We conduct a case study based on a real network scenario, and our experimental results demonstrate the superior performance of A2C-GS in terms of both efficiency and performance.

What problem does this paper attempt to address?

This paper attempts to solve the problem of network topology optimization. Specifically, network topology has a significant impact on important network performance indicators (such as link utilization, throughput, and latency), which is crucial for network operators. However, due to the combinatorial nature of network topology, it is very difficult to obtain the optimal solution, especially when network topology planning is often accompanied by specific management constraints. Although traditional manual tuning based on heuristic methods is widely used in practice, it cannot cover the global topology design space and cannot guarantee to find good solutions. To solve these problems, the author proposes a new algorithm based on deep reinforcement learning (DRL), called Advantage Actor Critic - Graph Searching (A2C - GS), for network topology optimization. A2C - GS contains three novel components: 1. **Validator**: Used to verify the correctness of the generated network topology. 2. **Graph Neural Network (GNN)**: Efficiently approximate the topology rating. 3. **DRL Action Layer**: Conduct topology search. Through these components, A2C - GS can efficiently search in a large - scale topology space and output topology structures with satisfactory performance. Experimental results show that A2C - GS is superior to existing heuristic search algorithms in both efficiency and performance. ### Formulas and Technical Details The main formulas and constraint conditions involved in the paper are as follows: #### Objective Function \[ \text{NetTopoOpt: } \max f(x) = U(x) + \gamma \text{Cost}(x, x_0) \] where: - \( U(x) \) represents the performance of the network under topology \( x \), such as link utilization. - \( \text{Cost}(x, x_0) \) represents the cost of transforming from the initial topology \( x_0 \) to the target topology \( x \). - \( \gamma \) is the weight between performance and cost, usually negative. #### Constraint Conditions 1. **Distance Feasibility**: \[ \text{Dist}(e) \leq D, \forall e \in E(x) \] where \( \text{Dist}(e) \) represents the distance between the end nodes of edge \( e \), and \( D \) is the maximum allowed connection distance. 2. **Link Utilization**: \[ \text{Load}(e) \leq L_{\text{max}}, \forall e \in E(x) \] where \( \text{Load}(e) \) represents the utilization of link \( e \), and \( L_{\text{max}} \) is the maximum allowed utilization level. 3. **Abstract Feasibility Requirement**: \[ M(x) = \text{True} \] This constraint is usually determined by network management requirements, allowing the operator to impose strategic restrictions on the final network topology. ### Method and Algorithm The A2C - GS algorithm contains three main parts: 1. **Representation Layer**: Learn the network objective function and compress the action and state spaces to reduce the computational complexity caused by the large - scale search space. 2. **DRL Agent**: Control actions based on learning of the A2C algorithm. 3. **Topology Validator**: Check the feasibility of the output topology to ensure that all network operator constraints are met. ### Experiments and Case Studies The paper demonstrates the effectiveness and performance of A2C - GS through a case study of a real - world network management scenario from China Mobile. Experimental results show that A2C - GS outperforms existing heuristic search algorithms in both small - scale and large - scale action spaces. In conclusion, by introducing the A2C - GS algorithm, this paper provides an efficient and effective method for network topology optimization, which can handle complex combinatorial optimization problems and nonlinear constraints.

Network Topology Optimization via Deep Reinforcement Learning

Enhancing Topology Optimization with Adaptive Deep Learning

GTD3-NET: A deep reinforcement learning-based routing optimization algorithm for wireless networks

A Topology Design Method for Satellite Networks Based on Deep Reinforcement Learning

Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case

Network Link Weight Optimization Based on Antisymmetric Deep Graph Networks and Reinforcement Learning

Controlling Directed Networks with Evolving Topologies

A Graph reinforcement learning based SDN routing path selection for optimizing long-term revenue

Routing Optimization With Deep Reinforcement Learning in Knowledge Defined Networking

GROM: A Generalized Routing Optimization Method with Graph Neural Network and Deep Reinforcement Learning

"Deep Reinforcement Learning for Engineering Design Through Topology Optimization of Elementally Discretized Design Domains"

Opti-DeepRoute: A Topology-Adaptive Deep Reinforcement Learning Based Service Provisioning Framework for Elastic Optical Network

Network planning with deep reinforcement learning

An Approach to Combine the Power of Deep Reinforcement Learning with a Graph Neural Network for Routing Optimization

Finding critical edges in networks through deep reinforcement learning

Deep Reinforcement Learning-Based Topology Optimization for Self-Organized Wireless Sensor Networks

Reconfigurable Network Topology Based on Deep Reinforcement Learning in Software-Defined Data-Center Networks

Graph Reinforcement Learning for Network Control via Bi-Level Optimization

Centroid-Guided Target-Driven Topology Control Method for UAV Ad-hoc Networks Based On Tiny Deep Reinforcement Learning Algorithm

A Routing Optimization Method for Software-Defined SGIN Based on Deep Reinforcement Learning

Decentralized Routing and Radio Resource Allocation in Wireless Ad Hoc Networks via Graph Reinforcement Learning