Abstract:Light field (LF) cameras suffer from a fundamental trade-off between spatial and angular resolutions. Additionally, due to the significant amount of data that needs to be recorded, the Lytro ILLUM, a modern LF camera, can only capture three frames per second. In this paper, we consider space-time super-resolution (SR) for LF videos, aiming at generating high-resolution and high-frame-rate LF videos from low-resolution and low-frame-rate observations. Extending existing space-time video SR methods to this task directly will meet two key challenges: 1) how to re-organize sub-aperture images (SAIs) efficiently and effectively given highly redundant LF videos, and 2) how to aggregate complementary information between multiple SAIs and frames considering the coherence in LF videos. To address the above challenges, we propose a novel framework for space-time super-resolving LF videos for the first time. First, we propose a novel Multi-Scale Dilated SAI Re-organization strategy for re-organizing SAIs into auxiliary view stacks with decreasing resolution as the Chebyshev distance in the angular dimension increases. In particular, the auxiliary view stack with original resolution preserves essential visual details, while the down-scaled view stacks capture long-range contextual information. Second, we propose the Multi-Scale Aggregated Feature extractor and the Angular-Assisted Feature Interpolation module to utilize and aggregate information from the spatial, angular, and temporal dimensions in LF videos. The former aggregates similar contents from different SAIs and frames for subsequent reconstruction in a disparity-free manner at the feature level, whereas the latter interpolates intermediate frames temporally by implicitly aggregating geometric information. Compared to other potential approaches, experimental results demonstrate that the reconstructed LF videos generated by our framework achieve higher reconstruction quality and better preserve the LF parallax structure and temporal consistency. The implementation code is available at https://github.com/zeyuxiao1997/LFSTVSR.

End-to-End Spatial-Angular Light Field Super-Resolution Using Parallax Structure Preservation Strategy.

Light field angular super-resolution based on structure and scene information

Light Field Reconstruction using Efficient Pseudo 4D Epipolar-Aware Structure

Light field angular super-resolution based on intrinsic and geometric information.

Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization

End-to-End Light Field Spatial Super-Resolution Network Using Multiple Epipolar Geometry

View Position Prior-Supervised Light Field Angular Super-Resolution Network with Asymmetric Feature Extraction and Spatial-Angular Interaction.

Piecewise Planar Super-Resolution for 3D Scene

Light Field Spatial Super-Resolution Via Geometric Feature Interaction

Efficient Light Field Angular Super-Resolution with Sub-Aperture Feature Learning and Macro-Pixel Upsampling.

Space-Time Super-Resolution for Light Field Videos

Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution.

Revisiting Spatio-Angular Trade-off in Light Field Cameras and Extended Applications in Super-Resolution

Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution

Light Field Spatial Super-resolution Via CNN Guided by A Single High-resolution RGB Image.

Light Field Image Super-Resolution Network Based on Angular Difference Enhancement

GLGNet: light field angular superresolution with arbitrary interpolation rates

Cross-Scale Reference-Based Light Field Super-Resolution

Geometry-assisted multi-representation view reconstruction network for Light Field image angular super-resolution

Light Field Image Super-Resolution Using Deformable Convolution

Research on super-resolution reconstruction of light field images based on spatial angle information decoupling