Abstract:Light field imaging is rapidly becoming an established method for generating flexible image based description of scene appearances. Compared to classical 2D imaging techniques, the angular information included in light fields enables effects such as post‐capture refocusing and the exploration of the scene from different vantage points. In this paper, we describe a novel GPU pipeline for compression and real‐time rendering of light field videos with full parallax. To achieve this, we employ a dictionary learning approach and train an ensemble of dictionaries capable of efficiently representing light field video data using highly sparse coefficient sets. A novel, key element in our representation is that we simultaneously compress both image data (pixel colors) and the auxiliary information (depth, disparity, or optical flow) required for view interpolation. During playback, the coefficients are streamed to the GPU where the light field and the auxiliary information are reconstructed using the dictionary ensemble and view interpolation is performed. In order to realize the pipeline we present several technical contributions including a denoising scheme enhancing the sparsity in the dataset which enables higher compression ratios, and a novel pruning strategy which reduces the size of the dictionary ensemble and leads to significant reductions in computational complexity during the encoding of a light field. Our approach is independent of the light field parameterization and can be used with data from any light field video capture system. To demonstrate the usefulness of our pipeline, we utilize various publicly available light field video datasets and discuss the medical application of documenting heart surgery.

A trillion frames per second: the techniques and applications of light-in-flight photography

Superluminal Motion-Assisted 4-Dimensional Light-in-Flight Imaging

From 3D to 5D: Computational light-in-flight imaging via an SPAD camera

Virtual Frame Technique: Ultrafast Imaging with Any Camera

Ultra-fast Lensless Computational Imaging through 5D Frequency Analysis of Time-resolved Light Transport

High-resolution single-shot ultrafast imaging at ten trillion frames per second

Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging

Ultrafast Imaging of Light Scattering Dynamics Using Second-Generation Compressed Ultrafast Photography

Flying with Photons: Rendering Novel Views of Propagating Light

High-speed videography of transparent media using illumination-based multiplexed schlieren

High-resolution 3D light-field imaging

A Gigapixel Computational Light-Field Camera

Event fields: Capturing light fields at high speed, resolution, and dynamic range

High Speed Photography: Techniques And Applications

Light-In-Flight Imaging by a Silicon Image Sensor: Toward the Theoretical Highest Frame Rate

Single-shot framing integration photography with high spatial resolution at 5.3*1012 frames per second by an inversed 4f system

Variable Aperture Light Field Photography: Overcoming the Diffraction-Limited Spatio-Angular Resolution Tradeoff.

Light Field Video Compression and Real Time Rendering

Continuously streaming compressed high-speed photography using time delay integration

Single-shot Real-Time Video Recording of a Photonic Mach Cone Induced by a Scattered Light Pulse

Investigation of single-shot high-speed photography based on spatial frequency multiplexing