Abstract:Optimization of user association in a densely deployed heterogeneous cellular network is usually challenging and even more complicated due to the dynamic nature of user mobility and fluctuation in user counts. While deep reinforcement learning (DRL) emerges as a promising solution, its application in practice is hindered by high trial-and-error costs in real world and unsatisfactory physical network performance during training. In addition, existing DRL-based user association methods are usually only applicable to scenarios with a fixed number of users due to convergence and compatibility challenges. In this paper, we propose a parallel digital twin (DT)-driven DRL method for user association and load balancing in networks with both dynamic user counts, distribution, and mobility patterns. Our method employs a distributed DRL strategy to handle varying user numbers and exploits a refined neural network structure for faster convergence. To address these DRL training-related challenges, we devise a high-fidelity DT construction technique, featuring a zero-shot generative user mobility model, named Map2Traj, based on a diffusion model. Map2Traj estimates user trajectory patterns and spatial distributions solely from street maps. Armed with this DT environment, DRL agents are enabled to be trained without the need for interactions with the physical network. To enhance the generalization ability of DRL models for dynamic scenarios, a parallel DT framework is further established to alleviate strong correlation and non-stationarity in single-environment training and improve the training efficiency. Numerical results show that the proposed parallel DT-driven DRL method achieves closely comparable performance to real environment training, and even outperforms those trained in a single real-world environment with nearly 20% gain in terms of cell-edge user performance.

Reinforcement Learning for Load-balanced Parallel Particle Tracing

Dynamic Data Repartitioning for Load-Balanced Parallel Particle Tracing

Dynamic Load Balancing Based On Constrained K-D Tree Decomposition For Parallel Particle Tracing

Load Balancing for Parallel Discrete Event Simulation of Stochastic Reaction and Diffusion

Optimal parallelization strategies for active flow control in deep reinforcement learning-based computational fluid dynamics

Adaptive Load Balancing of Parallel Applications with Multi-Agent Reinforcement Learning on Heterogeneous Systems

Efficient Parallel Reinforcement Learning Framework using the Reactor Model

Intelligent Decision-Making of Load Balancing Using Deep Reinforcement Learning and Parallel PSO in Cloud Environment

LBVis: Interactive Dynamic Load Balancing Visualization for Parallel Particle Tracing

Spreeze: High-Throughput Parallel Reinforcement Learning Framework

Asynchronous Parallel Reinforcement Learning for Optimizing Propulsive Performance in Fin Ray Control

Load Balancing for Ultradense Networks: A Deep Reinforcement Learning-Based Approach.

Reinforced Workload Distribution Fairness

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Accelerating Distributed Learning in Non-Dedicated Environments

Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks

A Deep Reinforcement Learning Approach to Efficient Distributed Optimization

MSRL: Distributed Reinforcement Learning with Dataflow Fragments

Reinforcement learning-trained optimisers and Bayesian optimisation for online particle accelerator tuning

SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores