Abstract:Metaverse has emerged as a revolutionary technique for transforming the way people interact with digital content, which relies on a distributed computing and communication infrastructure, encompassing terminal users, edge servers, and cloud servers. However, the rapid evolution of the Metaverse presents challenges that surpass the capabilities of existing communication and network infrastructures, particularly on network bandwidth and latency. Additionally, human experience becomes a critical factor in this domain. Therefore, we introduce a human-aware hierarchical software defined network (SDN) architecture consisting of a Metaverse cloud layer, a mobile edge computing (MEC) server empowered edge layer, and a distributed terminal layer. Each MEC server dynamically controls a multi-antenna base station (BS) and several reconfigurable intelligent surfaces (RISs) according to the terminal immersive experience requirements in real-time. To overcome the bandwidth limitation, we propose a novel smart reconfigurable spatial reuse new radio in unlicensed spectrum (NR-U) framework, which can realize customizable communications through flexibly and coordinately reconfiguring beams among the coordination between BSs and RISs. The objective function is formulated as a Lyapunov optimization based decentralized partially-observable Markov decision process (Dec-POMDP) problem to maximize the spectral efficiency while guaranteeing the latency and reliability requirements in Metaverse, via a joint user selection, phase-shift control, and beam coordination strategy. To solve the above non-convex, strongly coupled, and mixed integer nonlinear programming (MINLP), we propose a novel multi-agent hierarchical deep reinforcement learning (MAHDRL) algorithm that integrates deep Q-network (DQN) to solve discrete problems, deep deterministic policy gradient (DDPG) to solve continuous problems, and mixing network to capture complex interactions between multiple agents. Numerical results demonstrate the effectiveness of the proposed algorithm and verify the performance improvements compared to traditional multi-agent deep reinforcement learning (MADRL) algorithms.

Online Bipartite Matching for HAP Access in Space-Air-Ground Integrated Networks using Graph Neural Network-Enhanced Reinforcement Learning

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Deep Reinforcement Learning for Multi-User Access Control in Non-Terrestrial Networks.

Online Learning Based Joint Gateway Selection and User Scheduling in Non-Stationary Air-Ground Networks

Parallel Deep Reinforcement Learning Based Online User Association Optimization in Heterogeneous Networks.

Intelligent Access to Unlicensed Spectrum: A Mean Field Based Deep Reinforcement Learning Approach

HAPS-UAV-Enabled Heterogeneous Networks: A Deep Reinforcement Learning Approach

Tera-SpaceCom: GNN-based Deep Reinforcement Learning for Joint Resource Allocation and Task Offloading in TeraHertz Band Space Networks

Human-Aware Dynamic Hierarchical Network Control for Distributed Metaverse Services

Resilient Massive Access for SAGIN: A Deep Reinforcement Learning Approach

Primary-User-Friendly Dynamic Spectrum Anti-Jamming Access: A GAN-Enhanced Deep Reinforcement Learning Approach

Joint Interference Alignment and Power Control for Dense Networks Via Deep Reinforcement Learning

Offloading Strategy Based on Graph Neural Reinforcement Learning in Mobile Edge Computing

Network planning with deep reinforcement learning

A Multi-Agent Reinforcement Learning Approach for Massive Access in NOMA-URLLC Networks

Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks

Reinforcement Learning-Based Resource Allocation and Energy Efficiency Optimization for a Space–Air–Ground-Integrated Network

Joint Optimization of Handover Control and Power Allocation Based on Multi-Agent Deep Reinforcement Learning

User-Level Dynamic Beam Hopping Design for LEO Satellite Networks Based on Deep Reinforcement Learning Assisted Enhanced Genetic Algorithm

Neural Graph Matching Network: Learning Lawler’s Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matching

Autonomous Navigation and Configuration of Integrated Access Backhauling for UAV Base Station Using Reinforcement Learning