Abstract:Three-dimensional (3D) light field display, as a potential future display method, has attracted considerable attention. However, there still exist certain issues to be addressed, especially the capture of dense views in real 3D scenes. Using sparse cameras associated with view synthesis algorithm has become a practical method. Supervised convolutional neural network (CNN) is used to synthesize virtual views. However, such a large amount of training target views is sometimes difficult to be obtained and the training position is relatively fixed. Novel views can also be synthesized by unsupervised network MPVN, but the method has strict requirements on capturing multiple uniform horizontal viewpoints, which is not suitable in practice. Here, a method of dense-view synthesis based on unsupervised learning is presented, which can synthesize arbitrary virtual views with multiple free-posed views captured in the real 3D scene based on unsupervised learning. Multiple posed views are reprojected to the target position and input into the neural network. The network outputs a color tower and a selection tower indicating the scene distribution along the depth direction. A single image is yielded by the weighted summation of two towers. The proposed network is end-to-end trained based on unsupervised learning by minimizing errors during reconstructions of posed views. A virtual view can be predicted in a high quality by reprojecting posed views to the desired position. Additionally, a sequence of dense virtual views can be generated for 3D light-field display by repeated predictions. Experimental results demonstrate the validity of our proposed network. PSNR of synthesized views are around 30dB and SSIM are over 0.90. Since multiple cameras are supported to be placed in free-posed positions, there are not strict physical requirements and the proposed method can be flexibly used for the real scene capture. We believe this approach will contribute to the wide applications of 3D light-field display in the future.

Virtual view synthesis for 3D light-field display based on scene tower blending

Dense-view Synthesis for Three-Dimensional Light-Field Display Based on Unsupervised Learning.

Virtual view synthesis for 3D light-field display based on feature reprojection and fusion

Dense View Synthesis for Three-Dimensional Light-Field Display Based on Scene Geometric Reconstruction

Fast Virtual View Synthesis for an 8K 3D Light-Field Display Based on Cutoff-Nerf and 3D Voxel Rendering.

Multi-parallax views synthesis for three-dimensional light-field display using unsupervised CNN.

Dense view synthesis for three-dimensional light-field displays based on position-guiding convolutional neural network

Asymmetric Bidirectional View Synthesis for Free Viewpoint and Three-Dimensional Video

Virtual Viewpoint Generation for Three-Dimensional Display Based on the Compressive Light Field

Wide-baseline view synthesis for light-field display based on plane-depth-fused sweep volume

Real-time Representation and Rendering of High Resolution 3D Light Field Based on Texture Enhanced Optical Flow Prediction

View Synthesis with Multi-scale Cost Aggregation and Confidence Prior

A Novel Method of Multi-view Virtual Image Synthesis for Auto-stereoscopic Display

Virtual View Synthesis for Multi-view 3D Display

NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field.

Performance Improvement for Additive Light Field Displays with Weighted Simultaneous Algebra Reconstruction Technique and Tracked Views

Real-time High-resolution View Synthesis of Complex Scenes with Explicit 3D Visibility Reasoning

Real-time Light-Field Generation Based on the Visual Hull for the 3D Light-Field Display with Free-Viewpoint Texture Mapping

View Position Prior-Supervised Light Field Angular Super-Resolution Network with Asymmetric Feature Extraction and Spatial-Angular Interaction.

View Synthesis Using Foreground Object Extraction for Disparity Control and Image Inpainting.

Learning A Locally Unified 3D Point Cloud for View Synthesis