Abstract:As reconfigurable intelligent surfaces (RIS) emerge as a pivotal technology in the upcoming sixth-generation (6G) networks, their deployment within practical multiple operator (OP) networks presents significant challenges, including the coordination of RIS configurations among OPs, interference management, and privacy maintenance. A promising strategy is to treat RIS as a public resource managed by an RIS provider (RP), which can enhance resource allocation efficiency by allowing dynamic access for multiple OPs. However, the intricate nature of coordinating management and optimizing RIS configurations significantly complicates the implementation process. In this paper, we propose a hierarchical deep reinforcement learning (HDRL) approach that decomposes the complicated RIS resource optimization problem into several subtasks. Specifically, a top-level RP-agent is responsible for RIS allocation, while low-level OP-agents control their assigned RISs and handle beamforming, RIS phase-shifts, and user association. By utilizing the semi-Markov decision process (SMDP) theory, we establish a sophisticated interaction mechanism between the RP and OPs, and introduce an advanced hierarchical proximal policy optimization (HPPO) algorithm. Furthermore, we propose an improved sequential-HPPO (S-HPPO) algorithm to address the curse of dimensionality encountered with a single RP-agent. Experimental results validate the stability of the HPPO algorithm across various environmental parameters, demonstrating its superiority over other benchmarks for joint resource optimization. Finally, we conduct a detailed comparative analysis between the proposed S-HPPO and HPPO algorithms, showcasing that the S-HPPO algorithm achieves faster convergence and improved performance in large-scale RIS allocation scenarios.

Deep Reinforcement Learning Framework For Joint Resource Allocation In Heterogeneous Networks

Parallel Deep Reinforcement Learning Based Online User Association Optimization in Heterogeneous Networks.

Deep Reinforcement Learning Based Resource Allocation for Heterogeneous Networks

A Power Allocation Scheme Based on Deep Reinforcement Learning in HetNets.

Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Cellular Networks

Deep Reinforcement Learning for Energy-Efficient Power Control in Heterogeneous Networks

Distributed Deep Reinforcement Learning-Based Spectrum and Power Allocation for Heterogeneous Networks

DDPG with Transfer Learning and Meta Learning Framework for Resource Allocation in Underlay Cognitive Radio Network

Joint Interference Alignment and Power Control for Dense Networks Via Deep Reinforcement Learning

Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks

Mobility-Aware Centralized Reinforcement Learning for Dynamic Resource Allocation in HetNets

Deep Reinforcement Learning for Computation and Communication Resource Allocation in Multiaccess MEC Assisted Railway IoT Networks

DeepSlicing: Deep Reinforcement Learning Assisted Resource Allocation for Network Slicing

Deep Reinforcement Learning (DRL)-based Resource Management in Software-Defined and Virtualized Vehicular Ad Hoc Networks

Spectrum-efficient user grouping and resource allocation based on deep reinforcement learning for mmWave massive MIMO-NOMA systems

Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach

Deep Q-Network Based Resource Allocation for UAV-assisted Ultra-Dense Networks

The Next Generation Heterogeneous Satellite Communication Networks: Integration of Resource Management and Deep Reinforcement Learning.

A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks

Resource Allocation for Network Slicing in Dynamic Multi-Tenant Networks: A Deep Reinforcement Learning Approach