Abstract:Light field imaging is rapidly becoming an established method for generating flexible image based description of scene appearances. Compared to classical 2D imaging techniques, the angular information included in light fields enables effects such as post‐capture refocusing and the exploration of the scene from different vantage points. In this paper, we describe a novel GPU pipeline for compression and real‐time rendering of light field videos with full parallax. To achieve this, we employ a dictionary learning approach and train an ensemble of dictionaries capable of efficiently representing light field video data using highly sparse coefficient sets. A novel, key element in our representation is that we simultaneously compress both image data (pixel colors) and the auxiliary information (depth, disparity, or optical flow) required for view interpolation. During playback, the coefficients are streamed to the GPU where the light field and the auxiliary information are reconstructed using the dictionary ensemble and view interpolation is performed. In order to realize the pipeline we present several technical contributions including a denoising scheme enhancing the sparsity in the dataset which enables higher compression ratios, and a novel pruning strategy which reduces the size of the dictionary ensemble and leads to significant reductions in computational complexity during the encoding of a light field. Our approach is independent of the light field parameterization and can be used with data from any light field video capture system. To demonstrate the usefulness of our pipeline, we utilize various publicly available light field video datasets and discuss the medical application of documenting heart surgery.

4-D Epanechnikov Mixture Regression in Light Field Image Compression

5-D Epanechnikov Mixture-of-Experts in Light Field Image Compression

A Learning-Based Method Using Epipolar Geometry for Light Field Depth Estimation

Light Field Image Compression with Sub-apertures Reordering and Adaptive Reconstruction.

Three-dimensional Epanechnikov mixture regression in image coding

Depth Estimation Algorithm for Light Field Data by Epipolar Image Analysis and Region Interpolation

OPTIMIZED INTER-VIEW PREDICTION BASED LIGHT FIELD IMAGE COMPRESSION WITH ADAPTIVE RECONSTRUCTION

Learning Kernel-Modulated Neural Representation for Efficient Light Field Compression

Probabilistic-based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising

An Integrated Representation & Compression Scheme Based on Convolutional Autoencoders with 4D DCT Perceptual Encoding for High Dynamic Range Light Fields

High Efficiency Light Field Compression Via Virtual Reference And Hierarchical Mv-Hevc

Light Field Video Compression and Real Time Rendering

Light Field Compression With Disparity-Guided Sparse Coding Based on Structural Key Views

Light Field Image Compression Using Depth-Based Cnn In Intra Prediction

Light field image coding using a residual channel attention network–based view synthesis

Light Field Compression Based on Implicit Neural Representation

End-to-End Light Field Image Compression with Multi-Domain Feature Learning

Deep Decoding of $\ell_\infty$-coded Light Field Images

Evaluation of 4D Light Field Compression Methods

Data Compression of Light Field Using Multiscale Edges