Abstract:Future network services must adapt to the highly dynamic uplink and downlink traffic. To fulfill this requirement, the 3rd Generation Partnership Project (3GPP) proposed dynamic time division duplex (D-TDD) technology in Long Term Evolution (LTE) Release 11. Afterward, the 3GPP RAN#86 meeting clarified that 5G NR needs to support dynamic adjustment of the duplex pattern (transmission direction) in the time domain. Although 5G NR provides a more flexible duplex pattern, how to configure an effective duplex pattern according to services traffic is still an open research area. In this research, we propose a distributed multi-agent deep reinforcement learning (MARL) based decentralized D-TDD configuration method. First, we model a D-TDD configuration problem as a dynamic programming problem. Given the buffer length of all UE, we model the D-TDD configuration policy as a conditional probability distribution. Our goal is to find a D-TDD configuration policy that maximizes the expected discount return of all UE's sum rates. Second, in order to reduce signaling overhead, we design a fully decentralized solution with distributed MARL technology. Each agent in MARL makes decisions only based on local observations. We regard each base station (BS) as an agent, and each agent configures uplink and downlink time slot ratio according to length of intra-BS user (UE) queue buffer. Third, in order to solve the problem of overall system revenue caused by the lack of global information in MARL, we apply leniency control and binary LSTM (BLSTM) based auto-encoder. Leniency controller effectively controls Q-value estimation process in MARL according to Q-value and current network conditions, and auto-encoder makes up for the defect that leniency control cannot handle complex environments and high-dimensional data. Through the parallel distributed training, the global D-TDD policy is obtained. This method deploys the MARL algorithm on the Mobile Edge Computing (MEC) server of each BS and uses the storage and computing capabilities of the server for distributed training. The simulation results show that the proposed distributed MARL converges stably in various environments, and performs better than distributed deep reinforcement algorithm.

Dynamic uplink/downlink configuration using Q-learning in femtocell networks

Dynamic Femtocell Gnb On/Off Strategies and Seamless Dual Connectivity in 5G Heterogeneous Cellular Networks

Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control

QoS-aware energy-efficient power control in two-tier femtocell networks based on Q-learning

A Cooperative Q-learning Approach for Real-time Power Allocation in Femtocell Networks

Multi-Agent Reinforcement Learning Based Unlicensed Resource Sharing for LTE-U Networks.

Time-Division Cellular Networks with Full-Duplex Base Stations

A Distributed User Association Method for LTE-U by Enabling Q-Learning in Minority Game.

Joint Optimization of Fractional Frequency Reuse and Cell Clustering for Dynamic TDD Small Cell Networks

Analysis of Packet Throughput in Small Cell Networks under Clustered Dynamic TDD

Reinforcement Learning-Based Computing and Transmission Scheduling for LTE-U-Enabled IoT

Cell Selection in Two-Tier Femtocell Networks Using Q-learning Algorithm

Distributed interference management using Q-Learning in Cognitive Femtocell networks: New USRP-based Implementation

Joint User Pairing, Resource Block Allocation, And Power Control For Full-Duplex Cellular Networks

Multi-Agent Reinforcement Learning Based Fully Decentralized Dynamic Time Division Configuration for 5G and B5G Network

Subchannel Allocation Scheme for Two-tire Femtocell Networks Based on Q-learning

Power Allocation for Full-Duplex Communication Systems Based on Deep Deterministic Policy Gradient

Federated Deep Q-Learning and 5G load balancing

A Learning Approach for Traffic Offloading in Stochastic Heterogeneous Cellular Networks.

Joint Optimization of Switching Point and Power Control in Dynamic TDD Cell-Free Massive MIMO

Reinforcement Learning-Based Interference Control for Ultra-Dense Small Cells.