Abstract:In a cell-free wireless network, distributed access points (APs) jointly serve all user equipments (UEs) within their coverage area by using the same time/frequency resources. In this paper, we develop a novel downlink cell-free multiple-input multiple-output (MIMO) millimeter wave (mmWave) network architecture that enables all APs and UEs to dynamically self-partition into a set of independent cell-free subnetworks in a time-slot basis. For this, we propose several network partitioning algorithms based on deep reinforcement learning (DRL). Furthermore, to mitigate interference between different cell-free subnetworks, we develop a novel hybrid analog beamsteering-digital beamforming model that zero-forces interference among cell-free subnetworks and at the same time maximizes the instantaneous sum-rate of all UEs within each subnetwork. Specifically, the hybrid beamforming model is implemented by using a novel mixed DRL-convex optimization method in which analog beamsteering between APs and UEs is conducted based on DRL while digital beamforming is modeled and solved as a convex optimization problem. The DRL models for network clustering and hybrid beamsteering are combined into a single hierarchical DRL design that enables exchange of DRL agents' experiences during both network training and operation. We also benchmark the performance of DRL models for clustering and beamsteering in terms of network performance, convergence rate, and computational complexity. Results show a significant rate enhancement due to the proposed hybrid beamforming scheme compared to its conventional all-digital counterpart. This performance enhancement becomes more significant as the number of network partitions increases. For DRL-based network clustering, the policy gradient (PG) algorithm offers the best possible performance in terms of stability and convergence rate while the state-action-reward-state-action (SARSA) algorithm suffers from significant variance, slowe- convergence, and slightly inferior performance than other algorithms. For DRL-based beamsteering, the soft actor-critic (SAC) algorithm with continuous action space shows the best performance. Also, online training of the agents with varying channel state information (CSI) is observed to increase the variance of the Q-values and decrease the convergence rate, with no significant effect on the average reward. The simulation codes are available at: https://github.com/yasser-aleryani/mmWaveCellFree.git

Multi-agent Reinforcement Learning-based User Scheduling and Resource Allocation in Cell-free Networks.

Multi-Agent Reinforcement Learning for Multi-Cell Spectrum and Power Allocation

Distributed Adaptive Multiuser Scheduling Via Multi-Agent Reinforcement Learning in Multicell MIMO Cellular Networks

Multi-Agent Reinforcement Learning Based Unlicensed Resource Sharing for LTE-U Networks.

Traffic-driven Spectrum and Power Allocation Via Scalable Multi-Agent Reinforcement Learning

Power Allocation in Multi-User Cellular Networks: Deep Reinforcement Learning Approaches

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Joint Cooperative Clustering and Power Control for Energy-Efficient Cell-Free XL-MIMO with Multi-Agent Reinforcement Learning

Mobile Cell-Free Massive MIMO with Multi-Agent Reinforcement Learning: A Scalable Framework

Deep-Reinforcement-Learning-Based Scheduling with Contiguous Resource Allocation for Next-Generation Cellular Systems

Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Cellular Networks

Multiple Access in Dynamic Cell-Free Networks: Outage Performance and Deep Reinforcement Learning-Based Design

Multi-agent Reinforcement Learning for Energy Saving in Multi-Cell Massive MIMO Systems

Meta Federated Reinforcement Learning for Distributed Resource Allocation

A Self-Organizing Resource Allocation Strategy Based on Q-learning Approach in Ultra-Dense Networks

Multi-agent Deep Reinforcement Learning for Cross-Layer Scheduling in Mobile Ad-Hoc Networks

S-MFRL: Spiking Mean Field Reinforcement Learning for Dynamic Resource Allocation of D2D Networks

Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Design

Double-Layer Power Control for Mobile Cell-Free XL-MIMO with Multi-Agent Reinforcement Learning

Optimization of Energy Efficiency for Uplink Murllc over Multiple Cells Using Cooperative Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning Based Distributed Multi-User Scheduling and Beamforming Design in Multi-Cell Systems