Abstract:The task of novel view synthesis aims to generate unseen perspectives of an object or scene from a limited set of input images. Nevertheless, synthesizing novel views from a single image still remains a significant challenge in the realm of computer vision. Previous approaches tackle this problem by adopting mesh prediction, multi-plain image construction, or more advanced techniques such as neural radiance fields. Recently, a pre-trained diffusion model that is specifically designed for 2D image synthesis has demonstrated its capability in producing photorealistic novel views, if sufficiently optimized on a 3D finetuning task. Although the fidelity and generalizability are greatly improved, training such a powerful diffusion model requires a vast volume of training data and model parameters, resulting in a notoriously long time and high computational costs. To tackle this issue, we propose Efficient-3DiM, a simple but effective framework to learn a single-image novel-view synthesizer. Motivated by our in-depth analysis of the inference process of diffusion models, we propose several pragmatic strategies to reduce the training overhead to a manageable scale, including a crafted timestep sampling strategy, a superior 3D feature extractor, and an enhanced training scheme. When combined, our framework is able to reduce the total training time from 10 days to less than 1 day, significantly accelerating the training process under the same computational platform (one instance with 8 Nvidia A100 GPUs). Comprehensive experiments are conducted to demonstrate the efficiency and generalizability of our proposed method.

An Improved Novel View Synthesis Approach Based on Feature Fusion and Channel Attention.

Multi-Scale Feature Fusion for Single Image Novel View Synthesis

Novel View Synthesis for Large-scale Scene using Adversarial Loss.

Novel View Synthesis from a Single Unposed Image via Unsupervised Learning

Deep Learning based Novel View Synthesis

View Synthesis with Multi-scale Cost Aggregation and Confidence Prior

A Novel Intermediate View Synthesis Method

Novel View Synthesis with Pixel-Space Diffusion Models

Multi-View To Novel View: Synthesizing Novel Views With Self-Learned Confidence

Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

Depth-Assisted Full Resolution Network for Single Image-Based View Synthesis.

A neural refinement network for single image view synthesis

Content-Aware Warping for View Synthesis

Deep View Synthesis Via Self-Consistent Generative Network

CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency

Novel View Synthesis from Single Images via Point Cloud Transformation

ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis

Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

STrCNeRF: an Implicit Neural Representation Method Combining Global and Local Features from an Image for Novel View Synthesis

View Independent Generative Adversarial Network for Novel View Synthesis

Learning Novel View Synthesis from Heterogeneous Low-light Captures