Abstract:Optimization of user association in a densely deployed heterogeneous cellular network is usually challenging and even more complicated due to the dynamic nature of user mobility and fluctuation in user counts. While deep reinforcement learning (DRL) emerges as a promising solution, its application in practice is hindered by high trial-and-error costs in real world and unsatisfactory physical network performance during training. In addition, existing DRL-based user association methods are usually only applicable to scenarios with a fixed number of users due to convergence and compatibility challenges. In this paper, we propose a parallel digital twin (DT)-driven DRL method for user association and load balancing in networks with both dynamic user counts, distribution, and mobility patterns. Our method employs a distributed DRL strategy to handle varying user numbers and exploits a refined neural network structure for faster convergence. To address these DRL training-related challenges, we devise a high-fidelity DT construction technique, featuring a zero-shot generative user mobility model, named Map2Traj, based on a diffusion model. Map2Traj estimates user trajectory patterns and spatial distributions solely from street maps. Armed with this DT environment, DRL agents are enabled to be trained without the need for interactions with the physical network. To enhance the generalization ability of DRL models for dynamic scenarios, a parallel DT framework is further established to alleviate strong correlation and non-stationarity in single-environment training and improve the training efficiency. Numerical results show that the proposed parallel DT-driven DRL method achieves closely comparable performance to real environment training, and even outperforms those trained in a single real-world environment with nearly 20% gain in terms of cell-edge user performance.

Deep Reinforcement Learning-Based User Pairing In Full-Duplex Communication Systems

Wireless Channel Prediction for Multi-user Physical Layer with Deep Reinforcement Learning

Deep Reinforcement Learning for Joint Channel Selection and Power Control in D2D Networks

Power Allocation for Full-Duplex Communication Systems Based on Deep Deterministic Policy Gradient

Joint User Pairing, Resource Block Allocation, And Power Control For Full-Duplex Cellular Networks

Double Deep Q-Network Based Distributed Resource Matching Algorithm for D2D Communication

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems

A DRL Approach for RIS-Assisted Full-Duplex UL and DL Transmission: Beamforming, Phase Shift and Power Optimization

Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks

Dynamic Clustering and Resource Allocation Using Deep Reinforcement Learning for Smart-Duplex Networks

Joint Deep Reinforcement Learning and Unfolding: Beam Selection and Precoding for Mmwave Multiuser MIMO with Lens Arrays

An Autonomous Transmission Scheme Using Dueling DQN for D2D Communication Networks

Hybrid Centralized-Distributed Resource Allocation Based on Deep Reinforcement Learning for Cooperative D2D Communications

Deep Reinforcement Learning Based Dynamic Beam Selection in Dual-Band Communication Systems

A deep reinforcement learning-based D2D spectrum allocation underlaying a cellular network

User Pairing for Delay-Limited NOMA-Based Satellite Networks with Deep Reinforcement Learning

A Deep Learning-Based Transmission Scheme Using Reduced Feedback for D2D Networks

Deep Reinforcement Learning-Based Dynamic Spectrum Access for D2D Communication Underlay Cellular Networks

Deep Reinforcement Learning-aided Transmission Design for Energy-efficient Link Optimization in Vehicular Communications

User Association and Power Allocation for User-Centric Smart-Duplex Networks via Tree-Structured Deep Reinforcement Learning