Abstract:Metaverse has emerged as a revolutionary technique for transforming the way people interact with digital content, which relies on a distributed computing and communication infrastructure, encompassing terminal users, edge servers, and cloud servers. However, the rapid evolution of the Metaverse presents challenges that surpass the capabilities of existing communication and network infrastructures, particularly on network bandwidth and latency. Additionally, human experience becomes a critical factor in this domain. Therefore, we introduce a human-aware hierarchical software defined network (SDN) architecture consisting of a Metaverse cloud layer, a mobile edge computing (MEC) server empowered edge layer, and a distributed terminal layer. Each MEC server dynamically controls a multi-antenna base station (BS) and several reconfigurable intelligent surfaces (RISs) according to the terminal immersive experience requirements in real-time. To overcome the bandwidth limitation, we propose a novel smart reconfigurable spatial reuse new radio in unlicensed spectrum (NR-U) framework, which can realize customizable communications through flexibly and coordinately reconfiguring beams among the coordination between BSs and RISs. The objective function is formulated as a Lyapunov optimization based decentralized partially-observable Markov decision process (Dec-POMDP) problem to maximize the spectral efficiency while guaranteeing the latency and reliability requirements in Metaverse, via a joint user selection, phase-shift control, and beam coordination strategy. To solve the above non-convex, strongly coupled, and mixed integer nonlinear programming (MINLP), we propose a novel multi-agent hierarchical deep reinforcement learning (MAHDRL) algorithm that integrates deep Q-network (DQN) to solve discrete problems, deep deterministic policy gradient (DDPG) to solve continuous problems, and mixing network to capture complex interactions between multiple agents. Numerical results demonstrate the effectiveness of the proposed algorithm and verify the performance improvements compared to traditional multi-agent deep reinforcement learning (MADRL) algorithms.

MARL Sim2real Transfer: Merging Physical Reality With Digital Virtuality in Metaverse

S2rl

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation

Resource Allocation of Federated Learning Assisted Mobile Augmented Reality System in the Metaverse

A Scalable and Parallelizable Digital Twin Framework for Sustainable Sim2Real Transition of Multi-Agent Reinforcement Learning Systems

UAV-Assisted Dynamic Avatar Task Migration for Vehicular Metaverse Services: A Multi-Agent Deep Reinforcement Learning Approach

UAV aided Metaverse over Wireless Communications: A Reinforcement Learning Approach

Multiexperience-Assisted Efficient Multiagent Reinforcement Learning

Self-Sustaining Multiple Access with Continual Deep Reinforcement Learning for Dynamic Metaverse Applications

SC-MAIRL: Semi-Centralized Multi-Agent Imitation Reinforcement Learning

From Multi-agent to Multi-robot: A Scalable Training and Evaluation Platform for Multi-robot Reinforcement Learning

Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey

Multi-UAV Pursuit-Evasion with Online Planning in Unknown Environments by Deep Reinforcement Learning

Harnessing Online Knowledge Transfer for Enhanced Search and Rescue Decisions via Multi-Agent Reinforcement Learning

MARL-based Design of Multi-Unmanned Aerial Vehicle Assisted Communication System with Hybrid Gaming Mode

Multi-Agent Cooperative Reinforcement Learning in 3d Virtual World

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication

Human-Aware Dynamic Hierarchical Network Control for Distributed Metaverse Services