Abstract:This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple lowresolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learningbased approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

A Novel Light Field Super-Resolution Framework Based on Hybrid Imaging System.

Fast Image Super-Resolution for a Dual-Resolution Camera

Light field angular super-resolution based on intrinsic and geometric information.

Combining Exemplar-Based Approach and learning-Based Approach for Light Field Super-Resolution Using a Hybrid Imaging System

Light Field Image Super-Resolution Network Based on Angular Difference Enhancement

High-resolution Light Field Reconstruction Using a Hybrid Imaging System

Light field angular super-resolution based on structure and scene information

Unsupervised Learning of High-resolution Light Field Imaging via Beam Splitter-based Hybrid Lenses

Light Field Super-Resolution By Jointly Exploiting Internal and External Similarities

View Position Prior-Supervised Light Field Angular Super-Resolution Network with Asymmetric Feature Extraction and Spatial-Angular Interaction.

Multi-Dimension Fusion Network for Light Field Spatial Super-Resolution using Dynamic Filters

Light Field Super-Resolution Using Internal and External Similarities.

Breaking the Spatio-Angular Trade-off for Light Field Super-Resolution via LSTM Modelling on Epipolar Plane Images.

Super-Resolution Reconstruction of Light Field Images via Sparse Representation

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Deep Light Field Super-Resolution Using Frequency Domain Analysis and Semantic Prior

Light Field Spatial Super-Resolution Via Geometric Feature Interaction

Flexible Hybrid Lenses Light Field Super-Resolution Using Layered Refinement

Residual Networks for Light Field Image Super-Resolution

Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses

High-Resolution Light Field Cameras Based on A Hybrid Imaging System