Abstract:This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple lowresolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learningbased approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Three Dimensional Reconstruction Using a Lenslet Light Field Camera

Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion

Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Light Field Reconstruction using Efficient Pseudo 4D Epipolar-Aware Structure

LF-fusion: Dense and Accurate 3D Reconstruction from Light Field Images

A Coarse-to-Fine Convolutional Neural Network for Light Field Angular Super-Resolution

High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction

Learning Zero-Shot Dense Light Field Reconstruction from Heterogeneous Imaging

Light Field Angular Super-Resolution Via Dense Correspondence Field Reconstruction

Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction

Light field angular super-resolution based on structure and scene information

Flexible Light Field Angular Superresolution Via a Deep Coarse-to-Fine Framework

Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization

Light Field Image Super-Resolution Using Deformable Convolution

Dense reconstruction for sparsely-sampled light field

Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

Dense Light Field Reconstruction from Sparse Sampling Using Residual Network