Abstract:The next generation of networks will actively embrace artificial intelligence (AI) and machine learning (ML) technologies for automation networks and optimal network operation strategies. The emerging network structure represented by Open RAN (O-RAN) conforms to this trend, and the radio intelligent controller (RIC) at the centre of its specification serves as an ML applications host. Various ML models, especially Reinforcement Learning (RL) models, are regarded as the key to solving RAN-related multi-objective optimization problems. However, it should be recognized that most of the current RL successes are confined to abstract and simplified simulation environments, which may not directly translate to high performance in complex real environments. One of the main reasons is the modelling gap between the simulation and the real environment, which could make the RL agent trained by simulation ill-equipped for the real environment. This issue is termed as the sim2real gap. This article brings to the fore the sim2real challenge within the context of O-RAN. Specifically, it emphasizes the characteristics, and benefits that the digital twins (DT) could have as a place for model development and verification. Several use cases are presented to exemplify and demonstrate failure modes of the simulations trained RL model in real environments. The effectiveness of DT in assisting the development of RL algorithms is discussed. Then the current state of the art learning-based methods commonly used to overcome the sim2real challenge are presented. Finally, the development and deployment concerns for the RL applications realisation in O-RAN are discussed from the view of the potential issues like data interaction, environment bottlenecks, and algorithm design.

Deep reinforcement learning for RAN optimization and control

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

User Access Control in Open Radio Access Networks: A Federated Deep Reinforcement Learning Approach

Deep Reinforcement Learning for Energy Saving in Radio Access Network

Design and Evaluation of Deep Reinforcement Learning for Energy Saving in Open RAN

A Reinforcement Learning Framework for Autonomous Cell Activation and Customized Energy-Efficient Resource Allocation in C-RANs.

A Multi-Agent Deep Reinforcement Learning Approach for RAN Resource Allocation in O-RAN

CaRL: Cascade Reinforcement Learning with State Space Splitting for O-RAN based Traffic Steering

OpenRANet: Neuralized Spectrum Access by Joint Subcarrier and Power Allocation with Optimization-based Deep Learning

Learning with Side Information: Elastic Multi-Resource Control for the Open RAN

Collaborative Multi-BS Power Management for Dense Radio Access Network using Deep Reinforcement Learning

Intelligent O-RAN Traffic Steering for URLLC Through Deep Reinforcement Learning

Reinforcement Learning Based Dynamic Network Self-optimization for Heterogeneous Networks

Scalable Deep Reinforcement Learning for Routing and Spectrum Access in Physical Layer

Sim-to-Real Optimization of Complex Real World Mobile Network with Imperfect Information via Deep Reinforcement Learning from Self-play

Federated Deep Reinforcement Learning for User Access Control in Open Radio Access Networks

An Intelligent Routing Mechanism Based on Deep Reinforcement Learning

Computation Offloading in E-RAN Via Deep Reinforcement Learning

Deep Reinforcement Learning for 5G Networks: Joint Beamforming, Power Control, and Interference Coordination

Sim2real for Reinforcement Learning Driven Next Generation Networks