Abstract:We consider the reconstruction problem of video snapshot compressive imaging (SCI), which captures high-speed videos using a low-speed 2D sensor (detector). The underlying principle of SCI is to modulate sequential high-speed frames with different masks and then these encoded frames are integrated into a snapshot on the sensor and thus the sensor can be of low-speed. On one hand, video SCI enjoys the advantages of low-bandwidth, low-power and low-cost. On the other hand, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging and one of the bottlenecks lies in the reconstruction algorithm. Existing algorithms are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload. We first employ the image deep denoising priors to show that PnP can recover a UHD color video with 30 frames from a snapshot measurement. Since videos have strong temporal correlation, by employing the video deep denoising priors, we achieve a significant improvement in the results. Furthermore, we extend the proposed PnP algorithms to the color SCI system using mosaic sensors, where each pixel only captures the red, green or blue channels. A joint reconstruction and demosaicing paradigm is developed for flexible and high quality reconstruction of color video SCI systems. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm.

GAP-net for Snapshot Compressive Imaging

GAPMSF-Net: Generalized Alternating Projection With Multi-Stage Fusion Network for Snapshot Compressive Imaging

EiCSNet: Efficient Iterative Neural Network for Compressed Sensing Reconstruction

IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction

Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging

Deep learning for video compressive sensing

Plug-and-Play Algorithms for Video Snapshot Compressive Imaging

Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms

SCIGS: 3D Gaussians Splatting from a Snapshot Compressive Image

EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging

Snapshot Compressive Imaging: Principle, Implementation, Theory, Algorithms and Applications

Memory-Efficient Network for Large-scale Video Compressive Sensing

MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing

10-mega pixel snapshot compressive imaging with a hybrid coded aperture

Deep Optics for Video Snapshot Compressive Imaging

Spatial-Temporal Synergic Prior Driven Unfolding Network for Snapshot Compressive Imaging

Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging

Provable deep video denoiser using spatial-temporal information for video snapshot compressive imaging: Algorithm and convergence analysis

BIRNAT: Bidirectional Recurrent Neural Networks with Adversarial Training for Video Snapshot Compressive Imaging

Ten-mega-pixel snapshot compressive imaging with a hybrid coded aperture

Hybrid CNN-Transformer Architecture for Efficient Large-Scale Video Snapshot Compressive Imaging