Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models

Zeyu Yang,Zijie Pan,Chun Gu,Li Zhang

2024-10-02

Abstract:Recent advancements in 3D generation are predominantly propelled by improvements in 3D-aware image diffusion models. These models are pretrained on Internet-scale image data and fine-tuned on massive 3D data, offering the capability of producing highly consistent multi-view images. However, due to the scarcity of synchronized multi-view video data, it remains challenging to adapt this paradigm to 4D generation directly. Despite that, the available video and 3D data are adequate for training video and multi-view diffusion models separately that can provide satisfactory dynamic and geometric priors respectively. To take advantage of both, this paper presents Diffusion$^2$, a novel framework for dynamic 3D content creation that reconciles the knowledge about geometric consistency and temporal smoothness from these models to directly sample dense multi-view multi-frame images which can be employed to optimize continuous 4D representation. Specifically, we design a simple yet effective denoising strategy via score composition of pretrained video and multi-view diffusion models based on the probability structure of the target image array. To alleviate the potential conflicts between two heterogeneous scores, we further introduce variance-reducing sampling via interpolated steps, facilitating smooth and stable generation. Owing to the high parallelism of the proposed image generation process and the efficiency of the modern 4D reconstruction pipeline, our framework can generate 4D content within few minutes. Notably, our method circumvents the reliance on expensive and hard-to-scale 4D data, thereby having the potential to benefit from the scaling of the foundation video and multi-view diffusion models. Extensive experiments demonstrate the efficacy of our proposed framework in generating highly seamless and consistent 4D assets under various types of conditions.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem this paper attempts to address is several key challenges in dynamic 3D content generation, particularly how to efficiently generate highly consistent and spatiotemporally coherent 4D content without a large amount of synchronized multi-view video data. Specifically, the paper proposes a new framework called Diffusion2, which aims to optimize continuous 4D representations by directly sampling dense multi-view multi-frame image matrices through the combination of pre-trained video diffusion models and multi-view diffusion models.

### Main Problems:
1. **Lack of synchronized multi-view video data**: Existing 4D generation methods typically rely on large-scale synchronized multi-view video data, but such data is very scarce and difficult to obtain. This limits the development and application of 4D generation technology.
2. **Generation efficiency and stability**: Existing 4D generation methods (such as Score Distillation Sampling, SDS) have issues with slow generation speed and unstable optimization, especially when handling large-scale data.
3. **Geometric consistency and temporal smoothness**: 4D content generation needs to ensure both geometric consistency and temporal smoothness, which are two independent but closely related challenges.

### Solutions:
1. **Combining video diffusion models and multi-view diffusion models**: The paper proposes a new framework, Diffusion2, which directly generates multi-view multi-frame image matrices by combining the temporal smoothness priors of video diffusion models and the geometric consistency priors of multi-view diffusion models.
2. **Variational Reduced Sampling (VRS)**: To alleviate potential conflicts between different model scores, the paper introduces the Variational Reduced Sampling technique, which ensures the smoothness and consistency of the generated results by mixing different modes at high noise levels.
3. **Efficient reconstruction strategy**: The paper proposes a dynamic and texture-decoupled reconstruction strategy, converting the generated discrete image matrices into continuous 4D representations while minimizing detail loss caused by aggressive variational reduction strategies.

### Main Contributions:
1. **Proposed a new 4D generation framework**: Diffusion2 can generate highly consistent multi-view videos in a single reverse diffusion process without relying on any 4D datasets.
2. **Introduced Variational Reduced Sampling technique**: Effectively alleviated conflicts between different model scores, improving the smoothness and consistency of the generated results.
3. **Efficient reconstruction strategy**: By using a dynamic and texture-decoupled reconstruction method, it retains texture details of distant views and frames.
4. **Systematic experimental validation**: Experimental results show that Diffusion2 can generate high-quality 4D content under different input conditions and outperforms existing methods in terms of generation time and quality.

In summary, this paper addresses key challenges in 4D content generation through innovative methods and techniques, providing new ideas and solutions for the efficient generation of dynamic 3D content.