Abstract:In a cell-free wireless network, distributed access points (APs) jointly serve all user equipments (UEs) within their coverage area by using the same time/frequency resources. In this paper, we develop a novel downlink cell-free multiple-input multiple-output (MIMO) millimeter wave (mmWave) network architecture that enables all APs and UEs to dynamically self-partition into a set of independent cell-free subnetworks in a time-slot basis. For this, we propose several network partitioning algorithms based on deep reinforcement learning (DRL). Furthermore, to mitigate interference between different cell-free subnetworks, we develop a novel hybrid analog beamsteering-digital beamforming model that zero-forces interference among cell-free subnetworks and at the same time maximizes the instantaneous sum-rate of all UEs within each subnetwork. Specifically, the hybrid beamforming model is implemented by using a novel mixed DRL-convex optimization method in which analog beamsteering between APs and UEs is conducted based on DRL while digital beamforming is modeled and solved as a convex optimization problem. The DRL models for network clustering and hybrid beamsteering are combined into a single hierarchical DRL design that enables exchange of DRL agents' experiences during both network training and operation. We also benchmark the performance of DRL models for clustering and beamsteering in terms of network performance, convergence rate, and computational complexity. Results show a significant rate enhancement due to the proposed hybrid beamforming scheme compared to its conventional all-digital counterpart. This performance enhancement becomes more significant as the number of network partitions increases. For DRL-based network clustering, the policy gradient (PG) algorithm offers the best possible performance in terms of stability and convergence rate while the state-action-reward-state-action (SARSA) algorithm suffers from significant variance, slowe- convergence, and slightly inferior performance than other algorithms. For DRL-based beamsteering, the soft actor-critic (SAC) algorithm with continuous action space shows the best performance. Also, online training of the agents with varying channel state information (CSI) is observed to increase the variance of the Q-values and decrease the convergence rate, with no significant effect on the average reward. The simulation codes are available at: https://github.com/yasser-aleryani/mmWaveCellFree.git

Deep Q-Learning for Self-Organizing Networks Fault Management and Radio Performance Improvement

Safe Exploration in Wireless Security: A Safe Reinforcement Learning Algorithm With Hierarchical Structure

A Framework for Automated Cellular Network Tuning with Reinforcement Learning

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Toward Safe and Accelerated Deep Reinforcement Learning for Next-Generation Wireless Networks

DDPG with Transfer Learning and Meta Learning Framework for Resource Allocation in Underlay Cognitive Radio Network

Federated Deep Reinforcement Learning for User Access Control in Open Radio Access Networks

Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems

A Reinforcement Learning Approach for D2D Spectrum Sharing in Wireless Industrial URLLC Networks

Deep-Reinforcement-Learning-Based Intelligent Routing Strategy for FANETs

Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Cellular Networks

User Access Control in Open Radio Access Networks: A Federated Deep Reinforcement Learning Approach

Self-Organizing Cellular Radio Access Network with Deep Learning

AQMDRL: Automatic Quality of Service Architecture Based on Multistep Deep Reinforcement Learning in Software-Defined Networking

Application of deep neural network and deep reinforcement learning in wireless communication

Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Design

Distributed Learning Framework for eMBB-URLLC Multiplexing in Open Radio Access Networks

Energy Efficiency Optimization for Subterranean LoRaWAN Using A Reinforcement Learning Approach: A Direct-to-Satellite Scenario

Transfer Learning for Autonomous Cell Activation Based on Relational Reinforcement Learning With Adaptive Reward

Mobility Robustness Optimization for Self-Organizing Networks: A Deep Reinforcement Learning Approach

Intelligent User Association for Symbiotic Radio Networks Using Deep Reinforcement Learning