Abstract:In a cell-free wireless network, distributed access points (APs) jointly serve all user equipments (UEs) within their coverage area by using the same time/frequency resources. In this paper, we develop a novel downlink cell-free multiple-input multiple-output (MIMO) millimeter wave (mmWave) network architecture that enables all APs and UEs to dynamically self-partition into a set of independent cell-free subnetworks in a time-slot basis. For this, we propose several network partitioning algorithms based on deep reinforcement learning (DRL). Furthermore, to mitigate interference between different cell-free subnetworks, we develop a novel hybrid analog beamsteering-digital beamforming model that zero-forces interference among cell-free subnetworks and at the same time maximizes the instantaneous sum-rate of all UEs within each subnetwork. Specifically, the hybrid beamforming model is implemented by using a novel mixed DRL-convex optimization method in which analog beamsteering between APs and UEs is conducted based on DRL while digital beamforming is modeled and solved as a convex optimization problem. The DRL models for network clustering and hybrid beamsteering are combined into a single hierarchical DRL design that enables exchange of DRL agents' experiences during both network training and operation. We also benchmark the performance of DRL models for clustering and beamsteering in terms of network performance, convergence rate, and computational complexity. Results show a significant rate enhancement due to the proposed hybrid beamforming scheme compared to its conventional all-digital counterpart. This performance enhancement becomes more significant as the number of network partitions increases. For DRL-based network clustering, the policy gradient (PG) algorithm offers the best possible performance in terms of stability and convergence rate while the state-action-reward-state-action (SARSA) algorithm suffers from significant variance, slowe- convergence, and slightly inferior performance than other algorithms. For DRL-based beamsteering, the soft actor-critic (SAC) algorithm with continuous action space shows the best performance. Also, online training of the agents with varying channel state information (CSI) is observed to increase the variance of the Q-values and decrease the convergence rate, with no significant effect on the average reward. The simulation codes are available at: https://github.com/yasser-aleryani/mmWaveCellFree.git

A Machine Learning Approach for Beamforming in UDN Considering Selfish and Altruistic Balance.

Deep Reinforcement Learning for Distributed Dynamic Coordinated Beamforming in Massive MIMO Cellular Networks

Joint Deep Reinforcement Learning and Unfolding: Beam Selection and Precoding for Mmwave Multiuser MIMO with Lens Arrays

Deep Learning Based Joint Beam Selection and Precoding Design for Mmwave Systems with Lens Arrays

Beam Management in Ultra-Dense mmWave Network via Federated Reinforcement Learning: An Intelligent and Secure Approach

Deep Reinforcement Learning for Energy-Efficient Beamforming Design in Cell-Free Networks

A Robust Deep Learning-Based Beamforming Design for RIS-Assisted Multiuser MISO Communications with Practical Constraints

Dynamic Beam Pattern and Bandwidth Allocation Based on Multi-Agent Deep Reinforcement Learning for Beam Hopping Satellite Systems

Scalable Predictive Beamforming for IRS-Assisted Multi-User Communications: A Deep Learning Approach

Multi-objective Deep Reinforcement Learning Based Joint Beamforming and Power Allocation in UAV Assisted Cellular Communication

Deep Reinforcement Learning Coordinated Receiver Beamforming for Millimeter-Wave Train-ground Communications

Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Design

Distributed 3D-Beam Reforming for Hovering-Tolerant UAVs Communication over Coexistence: A Deep-Q Learning for Intelligent Space-Air-Ground Integrated Networks

Deep Learning Based Predictive Beamforming Design

Deep Reinforcement Learning Based Dynamic Beam Selection in Dual-Band Communication Systems

Scalable Predictive Beamforming for IRS-Assisted Multi-User Communications: A Deep Learning Approach

Traffic-Aware Hierarchical Beam Selection for Cell-Free Massive MIMO

Millimeter-Wave Networking in the Sky: A Machine Learning and Mean Field Game Approach for Joint Beamforming and Beam-Steering

Fast MIMO Beamforming via Deep Reinforcement Learning for High Mobility mmWave Connectivity

Deep Reinforcement Learning Based End-to-End Multiuser Channel Prediction and Beamforming

Active Beam Learning for Full-Duplex Wireless Systems