Abstract:The Depth-Image-Based Rendering (DIBR) algorithm is pivotal in the advancement of Virtual Reality (VR) and Augmented Reality (AR) technologies due to its capacity to generate virtual views from arbitrary perspectives. Nonetheless, the generation process is often marred by the occurrence of holes due to sharp depth transitions, significantly degrading the quality of the synthesized view. To mitigate this issue, this study introduces a layered hole-filling method to enhance the quality of virtual views. The effectiveness of our proposed method is ensured through three key techniques: Firstly, a depth-aware decomposition technique is employed to precisely segregate foreground objects from the background within a reference view. This is achieved by leveraging both the reference image and its corresponding depth map, facilitating accurate instance-level separation of foreground objects. Secondly, a Generative Adversarial Network (GAN)-enhanced background reconstruction technique is proposed to generate hole-free target views devoid of foreground objects. Lastly, the integration of Masked 3D Image Warping (M3DIW) and Layered Mergence (LM) algorithms facilitates filling holes with foreground or background textures in a layered manner. Comprehensive experimental results demonstrate the superiority of our proposed method compared to state-of-the-art methods. Notably, our method demonstrates an improvement of 7.5% in mean average Peak Signal-to-Noise Ratio (PSNR) and 1.8% in mean average Structural Similarity Index Measure (SSIM) compared to existing techniques. Additionally, it impressively lowers mean average Learned Perceptual Image Patch Similarity (LPIPS) by 28.8% and significantly reduces mean average Fréchet Inception Distance (FID) by 28.9% for all sequences tested. These results affirm the effectiveness of our approach in enhancing the quality of virtual view synthesis within DIBR applications. Source code is available at https://github.com/threedteam/dibr.

A NEW DEPTH-AIDED MULTIDIRECTIONAL DISOCCLUSION RESTORATION METHOD FOR DEPTH-IMAGE-BASED RENDERING

Depth-Aided Inpainting for Disocclusion Restoration of Multi-View Images Using Depth-Image-Based Rendering

A Novel Filling Disocclusion Method Based on Background Extraction in Depth-Image-Based-Rendering

A New Disocclusion Filling Approach in Depth Image Based Rendering for Stereoscopic Imaging

A New Multidirectional Extrapolation Hole-Filling Method For Depth-Image-Based Rendering

A New Method of DIBR Based on Background Inpainting

Depth-image-based Rendering with Spatial and Temporal Texture Synthesis for 3DTV

A Novel DIBR Method Based on Image Inpainting

Arbitrary view generation based on DIBR

A disocclusion filling method using multiple sprites with depth for virtual view synthesis

Hole-filling based on disparity map for DIBR.

Depth-image Based View Synthesis for Threedimensional Television

Adaptive 3D content generation via depth video based rendering

Arbitrary View Rendering Based on DIBR and Image Mergence

Hole-Filling for DIBR Based on Depth and Gradient Information

A Hardware Assisted Ldi Building Algorithm With Application To Hybrid Volume Rendering

Effective Removal of Artifacts from Views Synthesized Using Depth Image Based Rendering.

Layered Hole Filling Based on Depth-Aware Decomposition and GAN-Enhanced Background Reconstruction for DIBR

Stereoacuity-guided depth image based rendering

A Forward and Reverse Wrapping Depth Image-Based Rendering (FR-DIBR) Method for Arbitrary View Generation

A New Virtual View Rendering Method Based on Depth Image