Abstract:This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple lowresolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learningbased approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

End-to-end learned single lens design using fast differentiable ray tracing

4-K-resolution Minimalist Optical System Design Based on Deep Learning.

High-performance full-color imaging system based on end-to-end joint optimization of computer-generated holography and metalens.

End-to-end Learned Single Lens Design Using Improved Wiener Deconvolution

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

High-performance Imaging with an Advanced Non-Imaging Lens Based on Full-Path Optical Diffraction Calculation in Two-Dimensional Space

End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model

Curriculum Learning for ab initio Deep Learned Refractive Optics

All-day Thin-Lens Computational Imaging with Scene-Specific Learning Recovery

End-to-end complex lens design with differentiate ray tracing

Computational Diffraction-Limited Imaging with a Non-Imaging Spatial Information Transfer Lens

Automated design of freeform imaging systems

Lensless cameras using a mask based on almost perfect sequence through deep learning

Simulation Of Imaging And Refocusing For Light Field Photography

Optical design mode based on fast automatic design process for freeform reflective imaging systems with modest FOV

Unsupervised Learning of High-resolution Light Field Imaging via Beam Splitter-based Hybrid Lenses

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Simplified design method for optical imaging systems based on deep learning

Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses

Extended Ray-Mapping Method Based on Differentiable Ray-Tracing for Non-Paraxial and Off-Axis Freeform Illumination Lens Design.

Large depth-of-field ultra-compact microscope by progressive optimization and deep learning