UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving

Rui Chen,Zehuan Wu,Yichen Liu,Yuxin Guo,Jingcheng Ni,Haifeng Xia,Siyu Xia

2024-12-06

Abstract:The creation of diverse and realistic driving scenarios has become essential to enhance perception and planning capabilities of the autonomous driving system. However, generating long-duration, surround-view consistent driving videos remains a significant challenge. To address this, we present UniMLVG, a unified framework designed to generate extended street multi-perspective videos under precise control. By integrating single- and multi-view driving videos into the training data, our approach updates cross-frame and cross-view modules across three stages with different training objectives, substantially boosting the diversity and quality of generated visual content. Additionally, we employ the explicit viewpoint modeling in multi-view video generation to effectively improve motion transition consistency. Capable of handling various input reference formats (e.g., text, images, or video), our UniMLVG generates high-quality multi-view videos according to the corresponding condition constraints such as 3D bounding boxes or frame-level text descriptions. Compared to the best models with similar capabilities, our framework achieves improvements of 21.4% in FID and 36.5% in FVD.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to generate high - quality, multi - view - consistent and controllable long - term driving videos, so as to enhance the perception and planning capabilities of autonomous driving systems. Specifically, the existing generation algorithms have deficiencies in generating long - term, multi - view - consistent driving videos, and cannot meet the requirements such as long - term multi - view consistency, conditional controllability and diversity simultaneously. To solve these problems, the paper proposes a unified framework named UniMLVG, which aims to generate multi - view driving videos with high consistency and diversity by integrating single - view and multi - view driving video data and adopting a multi - task, multi - condition and multi - stage training strategy. In addition, UniMLVG can also generate high - quality multi - view videos according to different input reference formats (such as text, image or video), and support conditional constraints such as 3D bounding boxes or frame - level text descriptions. Compared with the existing technologies, UniMLVG has increased by 21.4% and 36.5% respectively in FID and FVD metrics, showing its significant advantages in generating high - quality, multi - view - consistent driving videos.