Abstract:Image-based geometric modeling and novel view synthesis based on sparse large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, the fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules (i.e., the attention-guided multi-scale residual fusion module, and the disparity field-guided deep re-regularization module), which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs, have been proposed. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis.

Extending Depth of Field by Varifocal Multi-view Computational Imaging for Metaverse

Multi-image-distance Imaging System for Extending Depth-of-field

Extending Depth of Field for Varifocal Multiview Images

Joint Focusing Method Based on Optimal Image-Distance Interval for Multi-Image-distance Imaging System

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering

Multi-channel Residual Network Model for Accurate Estimation of Spatially-Varying and Depth-Dependent Defocus Kernels.

Multi-focus image fusion: A Survey of the state of the art

Light Field All-in-focus Image Fusion Based on Edge Enhanced Guided Filtering

Light Field All-in-focus Image Fusion Based on Spatially-Guided Angular Information.

Federated Multi-View Synthesizing for Metaverse

Varifocal Multiview Images: Capturing and Visual Tasks

Mutli-focus image fusion based on guided filter and image matting network

Depth Extension For Multiple Focused Images By Adaptive Point Spread Function

MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image aided Generalizable Neural Radiance Field

Extending the depth of field of imaging systems using depth sensing camera

Fully Convolutional Network-Based Multifocus Image Fusion.

Multi-focus Image Fusion Using Fully Convolutional Two-stream Network for Visual Sensors.

Pixel Visibility Based Multifocus Image Fusion

Construction of all-in-focus images assisted by depth sensing

Fractal Dimension-Based Multi-Focus Image Fusion via Coupled Neural P Systems in NSCT Domain