Abstract:Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution as the limited detector resolution has to be shared with the angular dimension. LF spatial super-resolution (SR) thus becomes an indispensable part of the LF camera processing pipeline. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. The performance of existing methods is still limited as they fail to thoroughly explore the coherence among LF sub-aperture images (SAIs) and are insufficient in accurately preserving the scene's parallax structure. To tackle this challenge, we propose a novel learning-based LF spatial SR framework. Specifically, each SAI of an LF image is first coarsely and individually super-resolved by exploring the complementary information among SAIs with selective combinatorial geometry embedding. To achieve efficient and effective selection of the complementary information, we propose two novel sub-modules conducted hierarchically: the patch selector provides an option of retrieving similar image patches based on offline disparity estimation to handle large-disparity correlations; and the SAI selector adaptively and flexibly selects the most informative SAIs to improve the embedding efficiency. To preserve the parallax structure among the reconstructed SAIs, we subsequently append a consistency regularization network trained over a structure-aware loss function to refine the parallax relationships over the coarse estimation. In addition, we extend the proposed method to irregular LF data. To the best of our knowledge, this is the first learning-based SR method for irregular LF data. Experimental results over both synthetic and real-world LF datasets demonstrate the significant advantage of our approach over state-of-the-art methods.

Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion

Light Field Reconstruction with Arbitrary Angular Resolution Using a Deep Coarse-To-Fine Framework

Light Field Reconstruction With A Hybrid Sparse Regularization-Pseudo 4dcnn Framework

Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

Light Field Reconstruction Via Attention-Guided Deep Fusion of Hybrid Lenses

Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

Flexible Light Field Angular Superresolution Via a Deep Coarse-to-Fine Framework

LF-fusion: Dense and Accurate 3D Reconstruction from Light Field Images

Light Field Reconstruction using Efficient Pseudo 4D Epipolar-Aware Structure

A Coarse-to-Fine Convolutional Neural Network for Light Field Angular Super-Resolution

Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution

Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization

Learning Zero-Shot Dense Light Field Reconstruction from Heterogeneous Imaging

Adaptive Pixel Aggregation for Joint Spatial and Angular Super-Resolution of Light Field Images

Efficient Light Field Angular Super-Resolution with Sub-Aperture Feature Learning and Macro-Pixel Upsampling.

Learning a Compact Spatial-Angular Representation for Light Field

Sheared Epipolar Focus Spectrum for Dense Light Field Reconstruction.

Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction

Micro-Lens Image Stack Upsampling for Densely-Sampled Light Field Reconstruction

Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines

Light field angular super-resolution based on structure and scene information