Abstract:Novel view synthesis of remote sensing (RS) scenes is of great significance for scene visualization, human–computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for RS images due to its high complexity, view sparsity, and limited view-perspective variations. In this article, we propose a novel RS view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of RS images, we represent the 3-D space by combining implicit multiplane images (ImMPI) representation and deep neural networks. The 3-D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multiview input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as ImMPIs. To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of RS 3-D scenes and proposed a neural-network-based prior extractor to accelerate the optimization process. In addition, we propose a new dataset for RS novel view synthesis with multiview real-world Google Earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design.

Stereo Magnification: Learning View Synthesis using Multiplane Images

View Synthesis with Multi-scale Cost Aggregation and Confidence Prior

Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis

Stereo Vision Conversion from Planar Videos Based on Temporal Multiplane Images

View synthesis with multiplane images from computationally generated RGB-D light fields

Lightweight Multiplane Images Network for Real-Time Stereoscopic Conversion from Planar Video

Image mosaic based view synthesis for interactive stereoscopic display

Intermediate View Synthesis Based on Image Mosaic in Video Conferencing

Single-View View Synthesis with Self-Rectified Pseudo-Stereo

Multi-View Stereo Representation Revist: Region-Aware MVSNet

Remote Sensing Novel View Synthesis With Implicit Multiplane Representations

Stereoscopic Video Synthesis from A Monocular Video

Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction

MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds

SinMPI: Novel View Synthesis from a Single Image with Expanded Multiplane Images

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

View Synthesis from Multi-View RGB Data Using Multilayered Representation and Volumetric Estimation

Stereoscopic image-inpainting-based view synthesis algorithm for glasses-based and glasses-free 3D displays

Stereoscopic view synthesis with progressive structure reconstruction and scene constraints

Robust Multiview Synthesis for Wide-Baseline Camera Arrays

Multi-parallax views synthesis for three-dimensional light-field display using unsupervised CNN.