Abstract:The disaggregated and hierarchical architecture of advanced RAN presents significant challenges in efficiently placing baseband functions and user plane functions in conjunction with Multi-Access Edge Computing (MEC) to accommodate diverse 5G services. Therefore, this paper proposes a novel approach NetMind, which leverages Deep Reinforcement Learning (DRL) to determine the function placement strategies in RANs with diverse topologies, aiming at minimizing power consumption. NetMind formulates the function placement problem as a maze-solving task, enabling a Markov Decision Process with standardized action space scales across different networks. Additionally, a Graph Convolutional Network (GCN) based encoding mechanism is introduced, allowing features from different networks to be aggregated into a single RL agent. That facilitates the RL agent's generalization capability and minimizes the negative impact of retraining on power consumption. In an example with three sub-networks, NetMind achieves comparable performance to traditional methods that require a dedicated DRL agent for each network, resulting in a 70% reduction in training costs. Furthermore, it demonstrates a substantial 32.76% improvement in power savings and a 41.67% increase in service stability compared to benchmarks from the existing literature.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the issue of efficiently placing baseband functions and user plane functions (UPF) in advanced Radio Access Networks (RAN). Specifically, the paper focuses on optimizing the placement strategy of these functions with the support of Multi-access Edge Computing (MEC) to accommodate diverse 5G service demands while minimizing power consumption. ### Background and Challenges 1. **Challenges of Advanced RAN Architecture**: - The decoupled and layered architecture of advanced RAN brings significant challenges, especially in the efficient placement of baseband functions and user plane functions. - Traditional centralized RAN (C-RAN) solutions are no longer applicable. The new RAN architecture includes Radio Units (RU), Distributed Units (DU), Core Unit User Plane (CU-UP), and Core Unit Control Plane (CU-CP), which can be flexibly partitioned and deployed. 2. **Limitations of Existing Methods**: - **(M)ILP Methods**: Although they can achieve near-optimal results, they have high computational complexity and time consumption, making them unsuitable for latency-sensitive 5G services. - **DRL Methods**: Most existing DRL methods assume the environment is non-Markovian, meaning future states are independent of the current state, which may not hold theoretically. - **Static Network Architecture**: Existing research mainly focuses on static network architectures, ignoring the model generalization capability in different network topologies and the impact of retraining on power consumption performance. ### Solution To overcome the above issues, the paper proposes NetMind, a method based on Deep Reinforcement Learning (DRL) and introduces a Graph Convolutional Network (GCN) encoder to improve the model's adaptability and generalization capability. The main contributions of NetMind are as follows: 1. **Transforming the Baseband Function Placement Problem into a Maze Solving Problem**: - The baseband function placement problem is modeled as a Markov Decision Process (MDP), and the Deep Q-Learning (DQN) algorithm is used to optimize function placement and routing configuration to minimize power consumption. 2. **Introducing GCN Encoder and Decoder**: - The GCN encoder unifies the state information of different networks into a consistent format, thereby training a general DRL model applicable to networks with different structures, eliminating the cost of retraining in different scenarios. 3. **Reducing Training Costs**: - By training once and applying to multiple networks, NetMind's training cost is only 30% of training multiple networks individually. 4. **Performance Evaluation**: - Compared with benchmark methods such as random allocation, Greedy Heuristic Program (GHP), and Mixed Integer Linear Programming (MILP), NetMind shows excellent performance in power savings and service stability, achieving 32.76% power savings and 41.67% improvement in service stability. ### Conclusion NetMind effectively addresses the function placement problem in advanced RAN architecture by transforming the baseband function placement problem into a maze-solving problem and combining GCN encoder and DRL methods. This approach not only reduces training costs but also significantly improves power efficiency and service stability.

NetMind: Adaptive RAN Baseband Function Placement by GCN Encoding and Maze-solving DRL

Generalizable 5G RAN/MEC Slicing and Admission Control for Reliable Network Operation

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

DRL-based Energy-Efficient Baseband Function Deployments for Service-Oriented Open RAN

DDPG with Transfer Learning and Meta Learning Framework for Resource Allocation in Underlay Cognitive Radio Network

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

A Multi-Agent Deep Reinforcement Learning Approach for RAN Resource Allocation in O-RAN

RAN Slice Strategy Based on Deep Reinforcement Learning for Smart Grid

Collaborative Multi-BS Power Management for Dense Radio Access Network using Deep Reinforcement Learning

Design and Evaluation of Deep Reinforcement Learning for Energy Saving in Open RAN

Deep Reinforcement Learning for Computation and Communication Resource Allocation in Multiaccess MEC Assisted Railway IoT Networks

Intelligent Hybrid Resource Allocation in MEC-assisted RAN Slicing Network

DMADRL: A Distributed Multi-agent Deep Reinforcement Learning Algorithm for Cognitive Offloading in Dynamic MEC Networks

Intelligent Joint Network Slicing and Routing via GCN-powered Multi-Task Deep Reinforcement Learning

Priority-Aware Resource Allocation for RIS-assisted Mobile Edge Computing Networks: A Deep Reinforcement Learning Approach

A Reinforcement Learning Framework for Autonomous Cell Activation and Customized Energy-Efficient Resource Allocation in C-RANs.

Offloading Strategy Based on Graph Neural Reinforcement Learning in Mobile Edge Computing

Deep reinforcement learning for RAN optimization and control

Learning from Peers: Deep Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G RAN Slicing

Hierarchical Reinforcement Learning for RIS-Assisted Energy-Efficient RAN

Power Allocation in Multi-User Cellular Networks: Deep Reinforcement Learning Approaches