Abstract:Generating multi-view images based on text or single-image prompts is a critical capability for the creation of 3D content. Two fundamental questions on this topic are what data we use for training and how to ensure multi-view consistency. This paper introduces a novel framework that makes fundamental contributions to both questions. Unlike leveraging images from 2D diffusion models for training, we propose a dense consistent multi-view generation model that is fine-tuned from off-the-shelf video generative models. Images from video generative models are more suitable for multi-view generation because the underlying network architecture that generates them employs a temporal module to enforce frame consistency. Moreover, the video data sets used to train these models are abundant and diverse, leading to a reduced train-finetuning domain gap. To enhance multi-view consistency, we introduce a 3D-Aware Denoising Sampling, which first employs a feed-forward reconstruction module to get an explicit global 3D model, and then adopts a sampling strategy that effectively involves images rendered from the global 3D model into the denoising sampling loop to improve the multi-view consistency of the final images. As a by-product, this module also provides a fast way to create 3D assets represented by 3D Gaussians within a few seconds. Our approach can generate 24 dense views and converges much faster in training than state-of-the-art approaches (4 GPU hours versus many thousand GPU hours) with comparable visual quality and consistency. By further fine-tuning, our approach outperforms existing state-of-the-art methods in both quantitative metrics and visual effects. Our project page is

Multi-view Consistent Generative Adversarial Networks for Compositional 3D-Aware Image Synthesis

Multi-View Consistent Generative Adversarial Networks for 3D-Aware Image Synthesis

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models

Multi3D: 3D-Aware Multimodal Image Synthesis

Learning Multi-View Manifold for Single Image Based Modeling

Compositional GAN: Learning Conditional Image Composition

VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model

CC3D: Layout-Conditioned Generation of Compositional 3D Scenes

Compositional GAN: Learning Image-Conditional Binary Composition

NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images

View Independent Generative Adversarial Network for Novel View Synthesis

Multi-Constraint Transferable Generative Adversarial Networks for Cross-Modal Brain Image Synthesis

Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation

Deep View Synthesis Via Self-Consistent Generative Network

Towards Realistic 3D Embedding via View Alignment

3D-SSGAN: Lifting 2D Semantics for 3D-Aware Compositional Portrait Synthesis

MT-GAN: toward realistic image composition based on spatial features

3D-aware Image Generation and Editing with Multi-modal Conditions

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Consistent123: Improve Consistency for One Image to 3D Object Synthesis