SinMPI: Novel View Synthesis from a Single Image with Expanded Multiplane Images

Guo Pu,Peng-Shuai Wang,Zhouhui Lian

2023-12-18

Abstract:Single-image novel view synthesis is a challenging and ongoing problem that aims to generate an infinite number of consistent views from a single input image. Although significant efforts have been made to advance the quality of generated novel views, less attention has been paid to the expansion of the underlying scene representation, which is crucial to the generation of realistic novel view images. This paper proposes SinMPI, a novel method that uses an expanded multiplane image (MPI) as the 3D scene representation to significantly expand the perspective range of MPI and generate high-quality novel views from a large multiplane space. The key idea of our method is to use Stable Diffusion to generate out-of-view contents, project all scene contents into an expanded multiplane image according to depths predicted by monocular depth estimators, and then optimize the multiplane image under the supervision of pseudo multi-view data generated by a depth-aware warping and inpainting module. Both qualitative and quantitative experiments have been conducted to validate the superiority of our method to the state of the art. Our code and data are available at <a class="link-external link-https" href="https://github.com/TrickyGo/SinMPI" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the problem of novel view synthesis from a single image. Specifically, the researchers propose a new method called SinMPI, which utilizes extended multiplane images (MPI) to represent 3D scenes and generate high-quality, view-consistent new views from a single input image.

#### Main Contributions:
1. **Extended Multiplane Image Representation**: Compared to traditional MPI methods, SinMPI significantly enhances the ability to model 3D scenes by extending the view range of MPI, supporting a larger range of camera movements.
2. **Volume Rendering Optimization**: SinMPI uses MPI as a volumetric scene representation and fills in missing geometric information through volume rendering techniques, effectively mitigating the depth discretization and texture repetition issues brought by traditional MPI predictors.
3. **Extensive Experimental Validation**: The researchers conducted extensive experimental evaluations on multiple datasets, including the real-world LLFF dataset, NeRF synthetic dataset, DTU dataset, and COCO dataset. The experimental results demonstrate that SinMPI achieves state-of-the-art performance in novel view synthesis from a single image and exhibits excellent scene expansion capabilities.