Abstract:Light field imaging is rapidly becoming an established method for generating flexible image based description of scene appearances. Compared to classical 2D imaging techniques, the angular information included in light fields enables effects such as post‐capture refocusing and the exploration of the scene from different vantage points. In this paper, we describe a novel GPU pipeline for compression and real‐time rendering of light field videos with full parallax. To achieve this, we employ a dictionary learning approach and train an ensemble of dictionaries capable of efficiently representing light field video data using highly sparse coefficient sets. A novel, key element in our representation is that we simultaneously compress both image data (pixel colors) and the auxiliary information (depth, disparity, or optical flow) required for view interpolation. During playback, the coefficients are streamed to the GPU where the light field and the auxiliary information are reconstructed using the dictionary ensemble and view interpolation is performed. In order to realize the pipeline we present several technical contributions including a denoising scheme enhancing the sparsity in the dataset which enables higher compression ratios, and a novel pruning strategy which reduces the size of the dictionary ensemble and leads to significant reductions in computational complexity during the encoding of a light field. Our approach is independent of the light field parameterization and can be used with data from any light field video capture system. To demonstrate the usefulness of our pipeline, we utilize various publicly available light field video datasets and discuss the medical application of documenting heart surgery.

Light Field Image Compression Based on Quality Aware Pseudo-Temporal Sequence

Light Field Image Compression with Sub-apertures Reordering and Adaptive Reconstruction.

High Efficiency Light Field Compression Via Virtual Reference And Hierarchical Mv-Hevc

Pseudo-Sequence-Based Light Field Image Compression

Two-pass Light Field Image Compression for Spatial Quality and Angular Consistency.

Pseudo Sequence Based 2-D Hierarchical Coding Structure for Light-Field Image Compression

Shearlet Transform based Light Field Compression Under Low Bitrates

OPTIMIZED INTER-VIEW PREDICTION BASED LIGHT FIELD IMAGE COMPRESSION WITH ADAPTIVE RECONSTRUCTION

Light Field Image Compression Based on Deep Learning

Light Field Image Compression Using Multi-branch Spatial Transformer Networks Based View Synthesis

Light Field Image Compression Using Depth-Based Cnn In Intra Prediction

Light Field Compression Based on Implicit Neural Representation

Light Field Video Compression and Real Time Rendering

Effect of fatigue on the precision of a whole-body pointing task

Light Field Image Compression Using Generative Adversarial Network-Based View Synthesis

Data Compression of Light Field Using Multiscale Edges

4-D Epanechnikov Mixture Regression in Light Field Image Compression

Evaluation of 4D Light Field Compression Methods

An Integrated Representation & Compression Scheme Based on Convolutional Autoencoders with 4D DCT Perceptual Encoding for High Dynamic Range Light Fields

Light field image coding using a residual channel attention network–based view synthesis

Dynamic Light Field Compression Using Shared Fields and Region Blocks for Streaming Service.