Abstract:In the vehicular mixed reality (MR) Metaverse, the distance between physical and virtual entities can be overcome by fusing the physical and virtual environments with multi-dimensional communications in autonomous driving systems. Assisted by digital twin (DT) technologies, connected autonomous vehicles (AVs), roadside units (RSU), and virtual simulators can maintain the vehicular MR Metaverse via digital simulations for sharing data and making driving decisions collaboratively. However, large-scale traffic and driving simulation via realistic data collection and fusion from the physical world for online prediction and offline training in autonomous driving systems are difficult and costly. In this paper, we propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations for improving driving safety and traffic efficiency. First, we propose a multi-task DT offloading model for the reliable execution of heterogeneous DT tasks with different requirements at RSUs. Then, based on the preferences of AV's DTs and collected realistic data, virtual simulators can synthesize unlimited conditioned driving and traffic datasets to further improve robustness. Finally, we propose a multi-task enhanced auction-based mechanism to provide fine-grained incentives for RSUs in providing resources for autonomous driving. The property analysis and experimental results demonstrate that the proposed mechanism and architecture are strategy-proof and effective, respectively.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to use generative artificial intelligence (AI) to synthesize large - scale and condition - controllable traffic and driving data in the autonomous driving system in order to improve driving safety and traffic efficiency. Specifically, the paper proposes an autonomous driving architecture based on digital twin (DT) technology. This architecture can synthesize an unlimited amount of conditional traffic and driving data in the virtual space through generative AI for online prediction and offline training. In addition, the paper also proposes a multi - task - enhanced auction mechanism to provide fine - grained incentives to encourage roadside units (RSUs) to provide resource support for the autonomous driving system. ### Problems Solved by the Paper 1. **Difficulties in Large - Scale Real - Driving Data Collection and Fusion**: Directly collecting real - driving data on a large scale in the physical world is both difficult and expensive. The paper solves this problem by using generative AI to synthesize an unlimited amount of conditional driving and traffic data, thereby improving driving safety and traffic efficiency. 2. **Reliability of Multi - Task Digital Twin Task Offloading**: In order to improve the reliability of heterogeneous digital twin task execution for different requirements, the paper proposes a multi - task digital twin task offloading model, allowing autonomous vehicles (AVs) to offload multiple tasks to RSUs for real - time execution. 3. **Data Synthesis in Traffic and Driving Simulation**: The paper uses a generative AI - enhanced virtual simulator to synthesize large - scale and conditional driving data according to the perception data of the physical world and user preferences, further enhancing the robustness of driving decisions. 4. **Resource Incentive Mechanism**: In order to encourage RSUs to provide communication and computing resources to support the autonomous driving system, the paper proposes a multi - task - enhanced auction mechanism, providing fine - grained allocation results and prices to ensure the effective use of resources. ### Main Contributions - Proposing a mixed - reality (MR) meta - universe architecture based on digital twin technology. This architecture uses generative AI to synthesize large - scale and conditional traffic and driving data, improving driving safety and traffic efficiency. - Designing a reliable multi - task digital twin task offloading framework, allowing AVs to continuously offload multiple tasks with different requirements to RSUs and update the digital twin in the virtual space. - Using a generative AI - enhanced virtual simulator to synthesize new driving data for AVs' decision - making and training. - Proposing a multi - task - enhanced auction mechanism to provide fine - grained incentives to encourage RSUs to provide resource support for the autonomous driving system and ensure the effective use of resources. Through these methods, the paper aims to build an efficient, reliable and safe autonomous driving ecosystem and promote the development and application of autonomous driving technology.

Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

Hybrid Operations of Human Driving Vehicles and Automated Vehicles with Data-Driven Agent-Based Simulation

Interactive Critical Scenario Generation for Autonomous Vehicles Testing Based on In-depth Crash Data Using Reinforcement Learning

Leveraging Digital Twin and DRL for Collaborative Context Offloading in C-V2X Autonomous Driving

Multi-Agent Deep Reinforcement Learning for Dynamic Avatar Migration in AIoT-enabled Vehicular Metaverses with Trajectory Prediction

Real-World Data Inspired Interactive Connected Traffic Scenario Generation

SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model

Digital Twin and Artificial Intelligence-Empowered Panoramic Video Streaming: Reducing Transmission Latency in the Extended Reality-Assisted Vehicular Metaverse

Towards the Vehicular Metaverse: Exploring Distributed Inference with Transformer-Based Diffusion Model

Data-driven Traffic Simulation: A Comprehensive Review

Data Generation Using Simulation Technology to Improve Perception Mechanism of Autonomous Vehicles

Parallel Driving with Big Models and Foundation Intelligence in Cyber-Physical-Social Spaces

An AR-Based Meta Vehicle Road Cooperation Testing Systems: Framework, Components Modeling and an Implementation Example

Scalable Traffic Simulation for Autonomous Driving Via Multi-Agent Goal Assignment and Autoregressive Goal-Directed Planning

Scenario Generation for Autonomous Vehicles with Deep-Learning-Based Heterogeneous Driver Models: Implementation and Verification

Hybrid-Generative Diffusion Models for Attack-Oriented Twin Migration in Vehicular Metaverses

Generative Diffusion-based Contract Design for Efficient AI Twins Migration in Vehicular Embodied AI Networks

Generating Critical Scenarios for Testing Automated Driving Systems

TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Behaviors for Autonomous Driving

MetaTrading: An Immersion-Aware Model Trading Framework for Vehicular Metaverse Services