Abstract:Optimization of user association in a densely deployed heterogeneous cellular network is usually challenging and even more complicated due to the dynamic nature of user mobility and fluctuation in user counts. While deep reinforcement learning (DRL) emerges as a promising solution, its application in practice is hindered by high trial-and-error costs in real world and unsatisfactory physical network performance during training. In addition, existing DRL-based user association methods are usually only applicable to scenarios with a fixed number of users due to convergence and compatibility challenges. In this paper, we propose a parallel digital twin (DT)-driven DRL method for user association and load balancing in networks with both dynamic user counts, distribution, and mobility patterns. Our method employs a distributed DRL strategy to handle varying user numbers and exploits a refined neural network structure for faster convergence. To address these DRL training-related challenges, we devise a high-fidelity DT construction technique, featuring a zero-shot generative user mobility model, named Map2Traj, based on a diffusion model. Map2Traj estimates user trajectory patterns and spatial distributions solely from street maps. Armed with this DT environment, DRL agents are enabled to be trained without the need for interactions with the physical network. To enhance the generalization ability of DRL models for dynamic scenarios, a parallel DT framework is further established to alleviate strong correlation and non-stationarity in single-environment training and improve the training efficiency. Numerical results show that the proposed parallel DT-driven DRL method achieves closely comparable performance to real environment training, and even outperforms those trained in a single real-world environment with nearly 20% gain in terms of cell-edge user performance.

Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies

DeepThrottle: Deep Reinforcement Learning for Router Throttling to Defend Against DDoS Attack in SDN

Towards Real-Time Routing Optimization with Deep Reinforcement Learning: Open Challenges

Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks

Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case

Distributed Deep Reinforcement Learning for Resource Allocation in Digital Twin Networks

DNLight: Learning Efficient Evaluation for Traffic Signal Control

Enabling efficient routing for traffic engineering in SDN with Deep Reinforcement Learning

Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges

A Deep Reinforcement Learning Approach for Adaptive Traffic Routing in Next-gen Networks

Experience-driven Networking: A Deep Reinforcement Learning based Approach

GTD3-NET: A deep reinforcement learning-based routing optimization algorithm for wireless networks

Distributed and Adaptive Traffic Engineering with Deep Reinforcement Learning

Optimizing Deep-Space DTN Congestion Control Via Deep Reinforcement Learning

Routing Optimization With Deep Reinforcement Learning in Knowledge Defined Networking

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

TITE: A Transformer-Based Deep Reinforcement Learning Approach for Traffic Engineering in Hybrid SDN with Dynamic Traffic

Sim-to-Real Optimization of Complex Real World Mobile Network with Imperfect Information via Deep Reinforcement Learning from Self-play

Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing

On Deep Reinforcement Learning for Static Routing and Wavelength Assignment