Video Snapshot Compressive Imaging Via Deep Learning with Adversarial Training

Honghao Huang,Chengyang Hu,Sigang Yang,Minghua Chen,Hongwei Chen

DOI: https://doi.org/10.1364/cleo_at.2022.jw3b.172

2022-01-01

Abstract:We present deep learning with adversarial training for video compressive imaging, which can recover 50 frames from a snapshot with boosted visual quality compared to typical iterative algorithm and neural network with simple loss function.

What problem does this paper attempt to address?

Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement

Jianing Deng,Li Wang,Shiliang Pu,Cheng Zhuo

DOI: https://doi.org/10.1609/aaai.v34i07.6697

2020-01-01

Abstract:Recent years have witnessed remarkable success of deep learning methods in quality enhancement for compressed video. To better explore temporal information, existing methods usually estimate optical flow for temporal motion compensation. However, since compressed video could be seriously distorted by various compression artifacts, the estimated optical flow tends to be inaccurate and unreliable, thereby resulting in ineffective quality enhancement. In addition, optical flow estimation for consecutive frames is generally conducted in a pairwise manner, which is computational expensive and inefficient. In this paper, we propose a fast yet effective method for compressed video quality enhancement by incorporating a novel Spatio-Temporal Deformable Fusion (STDF) scheme to aggregate temporal information. Specifically, the proposed STDF takes a target frame along with its neighboring reference frames as input to jointly predict an offset field to deform the spatio-temporal sampling positions of convolution. As a result, complementary information from both target and reference frames can be fused within a single Spatio-Temporal Deformable Convolution (STDC) operation. Extensive experiments show that our method achieves the state-of-the-art performance of compressed video quality enhancement in terms of both accuracy and efficiency.
BIRNAT: Bidirectional Recurrent Neural Networks with Adversarial Training for Video Snapshot Compressive Imaging

Ziheng Cheng,Ruiying Lu,Zhengjue Wang,Hao Zhang,Bo Chen,Ziyi Meng,Xin Yuan

DOI: https://doi.org/10.1007/978-3-030-58586-0_16

2020-01-01

Abstract:We consider the problem of video snapshot compressive imaging (SCI), where multiple high-speed frames are coded by different masks and then summed to a single measurement. This measurement and the modulation masks are fed into our Recurrent Neural Network (RNN) to reconstruct the desired high-speed frames. Our end-to-end sampling and reconstruction system is dubbed BIdirectional Recurrent Neural networks with Adversarial Training (BIRNAT). To our best knowledge, this is the first time that recurrent networks are employed to SCI problem. Our proposed BIRNAT outperforms other deep learning based algorithms and the state-of-the-art optimization based algorithm, DeSCI, through exploiting the underlying correlation of sequential video frames. BIRNAT employs a deep convolutional neural network with Resblock and feature map self-attention to reconstruct the first frame, based on which bidirectional RNN is utilized to reconstruct the following frames in a sequential manner. To improve the quality of the reconstructed video, BIRNAT is further equipped with the adversarial training besides the mean square error loss. Extensive results on both simulation and real data (from two SCI cameras) demonstrate the superior performance of our BIRNAT system. The codes are available at https://github.com/BoChenGroup/BIRNAT.
Deep Learning for Snapshot Image Compressive Sensing

Haomiao Zhang,Ping Wang,Xin Yuan

DOI: https://doi.org/10.1364/3d.2023.jw2a.11

2023-01-01

Abstract:This work proposes a novel image compression approach by spatially multiplexing the non-local information of image, which is a potential implementation of snapshot image compressive sensing.
Deep learning for video compressive sensing

Mu Qiao,Ziyi Meng,Jiawei Ma,Xin Yuan

DOI: https://doi.org/10.1063/1.5140721

IF: 5.6

2020-03-01

APL Photonics

Abstract:We investigate deep learning for video compressive sensing within the scope of snapshot compressive imaging (SCI). In video SCI, multiple high-speed frames are modulated by different coding patterns and then a low-speed detector captures the integration of these modulated frames. In this manner, each captured measurement frame incorporates the information of all the coded frames, and reconstruction algorithms are then employed to recover the high-speed video. In this paper, we build a video SCI system using a digital micromirror device and develop both an end-to-end convolutional neural network (E2E-CNN) and a Plug-and-Play (PnP) framework with deep denoising priors to solve the inverse problem. We compare them with the iterative baseline algorithm GAP-TV and the state-of-the-art DeSCI on real data. Given a determined setup, a well-trained E2E-CNN can provide video-rate high-quality reconstruction. The PnP deep denoising method can generate decent results without task-specific pre-training and is faster than conventional iterative algorithms. Considering speed, accuracy, and flexibility, the PnP deep denoising method may serve as a baseline in video SCI reconstruction. To conduct quantitative analysis on these reconstruction algorithms, we further perform a simulation comparison on synthetic data. We hope that this study contributes to the applications of SCI cameras in our daily life.

optics,physics, applied
Deep Optics for Video Snapshot Compressive Imaging

Ping Wang,Lishun Wang,Xin Yuan

DOI: https://doi.org/10.1109/ICCV51070.2023.00977

2024-04-08

Abstract:Video snapshot compressive imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector, whose backbones rest in optical modulation patterns (also known as masks) and a computational reconstruction algorithm. Advanced deep learning algorithms and mature hardware are putting video SCI into practical applications. Yet, there are two clouds in the sunshine of SCI: i) low dynamic range as a victim of high temporal multiplexing, and ii) existing deep learning algorithms' degradation on real system. To address these challenges, this paper presents a deep optics framework to jointly optimize masks and a reconstruction network. Specifically, we first propose a new type of structural mask to realize motion-aware and full-dynamic-range measurement. Considering the motion awareness property in measurement domain, we develop an efficient network for video SCI reconstruction using Transformer to capture long-term temporal dependencies, dubbed Res2former. Moreover, sensor response is introduced into the forward model of video SCI to guarantee end-to-end model training close to real system. Finally, we implement the learned structural masks on a digital micro-mirror device. Experimental results on synthetic and real data validate the effectiveness of the proposed framework. We believe this is a milestone for real-world video SCI. The source code and data are available at

Computer Vision and Pattern Recognition
Hybrid CNN-Transformer Architecture for Efficient Large-Scale Video Snapshot Compressive Imaging

Miao Cao,Lishun Wang,Mingyu Zhu,Xin Yuan

DOI: https://doi.org/10.1007/s11263-024-02101-y

IF: 13.369

2024-05-21

International Journal of Computer Vision

Abstract:Video snapshot compressive imaging (SCI) uses a low-speed 2D detector to capture high-speed scene, where the dynamic scene is modulated by different masks and then compressed into a snapshot measurement. Following this, a reconstruction algorithm is needed to reconstruct the high-speed video frames. Although state-of-the-art (SOTA) deep learning-based reconstruction algorithms have achieved impressive results, they still face the following challenges due to excessive model complexity and GPU memory limitations: (1) These models need high computational cost, and (2) They are usually unable to reconstruct large-scale video frames at high compression ratios. To address these issues, we develop an efficient network for video SCI by using hierarchical residual-like connections and hybrid CNN-Transformer structure within a single residual block, dubbed EfficientSCI++ . The EfficientSCI++ network can well explore spatial-temporal correlation using convolution in the spatial domain and Transformer in the temporal domain , respectively. We are the first time to demonstrate that a UHD color video ( ) with high compression ratio (40) can be reconstructed from a snapshot 2D measurement using a single end-to-end deep learning model with PSNR above 34 dB. Moreover, a mixed-precision model is trained to further accelerate the video SCI reconstruction process and save memory footprint. Extensive results on both simulation and real data demonstrate that, compared with precious SOTA methods, our proposed EfficientSCI++ and EfficientSCI can achieve comparable reconstruction quality with much cheaper computational cost and better real-time performance. Code is available at https://github.com/mcao92/EfficientSCI-plus-plus.

computer science, artificial intelligence
Key frames assisted hybrid encoding for photorealistic compressive video sensing

Honghao Huang,Jiajie Teng,Liang Yu,Chengyang Hu,Minghua Chen,Yang Shen,Hongwei Chen

DOI: https://doi.org/10.48550/arxiv.2207.12627

2022-01-01

Abstract:Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct photorealistic video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.
EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging

Lishun Wang,Miao Cao,Xin Yuan

2023-05-18

Abstract:Video snapshot compressive imaging (SCI) uses a two-dimensional detector to capture consecutive video frames during a single exposure time. Following this, an efficient reconstruction algorithm needs to be designed to reconstruct the desired video frames. Although recent deep learning-based state-of-the-art (SOTA) reconstruction algorithms have achieved good results in most tasks, they still face the following challenges due to excessive model complexity and GPU memory limitations: 1) these models need high computational cost, and 2) they are usually unable to reconstruct large-scale video frames at high compression ratios. To address these issues, we develop an efficient network for video SCI by using dense connections and space-time factorization mechanism within a single residual block, dubbed EfficientSCI. The EfficientSCI network can well establish spatial-temporal correlation by using convolution in the spatial domain and Transformer in the temporal domain, respectively. We are the first time to show that an UHD color video with high compression ratio can be reconstructed from a snapshot 2D measurement using a single end-to-end deep learning model with PSNR above 32 dB. Extensive results on both simulation and real data show that our method significantly outperforms all previous SOTA algorithms with better real-time performance. The code is at <a class="link-external link-https" href="https://github.com/ucaswangls/EfficientSCI.git" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Computation and Language,Image and Video Processing
Compressed Video Sensing Based on Deep Generative Adversarial Network

DOI: https://doi.org/10.1007/s00034-024-02672-8

IF: 2.311

2024-05-10

Circuits Systems and Signal Processing

Abstract:This paper considers the deep-learning-aided compressed video sensing problem. To this end, a deep generative adversarial network has been proposed to provide an approximation of the non-reference frame using its corresponding reference frame. The tests confirm the superiority of this scheme over the conventional methods used earlier. Furthermore, two scenarios have been suggested for deep compressed video sensing and recovery. In the first scenario, the difference between the non-reference frame and its approximation obtained from the pre-trained network is compressively sampled and transmitted to the receiver where the proposed residual reconstruction network is adopted to reconstruct the signal. The second scenario utilizes a pre-trained network followed by an augmented layer to approximate the non-reference frames. In the transmitter, the parameters of the augmented layer are trained for the current non-reference block. Instead of transmitting the samples of the block, the parameters of its trained augmented layer are sent to the receiver where the reconstruction is done using the same pre-trained network. The performances of the proposed scenarios demonstrate their objective and subjective superiority over the state-of-the-art algorithms in both the reconstruction quality and run time.

engineering, electrical & electronic
Recurrent Neural Networks for Snapshot Compressive Imaging

Ziheng Cheng,Bo Chen,Ruiying Lu,Zhengjue Wang,Hao Zhang,Ziyi Meng,Xin Yuan

DOI: https://doi.org/10.1109/tpami.2022.3161934

IF: 23.6

2022-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Conventional high-speed and spectral imaging systems are expensive and they usually consume a significant amount of memory and bandwidth to save and transmit the high-dimensional data. By contrast, snapshot compressive imaging (SCI), where multiple sequential frames are coded by different masks and then summed to a single measurement, is a promising idea to use a 2-dimensional camera to capture 3-dimensional scenes. In this paper, we consider the reconstruction problem in SCI, i.e., recovering a series of scenes from a compressed measurement. Specifically, the measurement and modulation masks are fed into our proposed network, dubbed BIdirectional Recurrent Neural networks with Adversarial Training (BIRNAT) to reconstruct the desired frames. BIRNAT employs a deep convolutional neural network with residual blocks and self-attention to reconstruct the first frame, based on which a bidirectional recurrent neural network is utilized to sequentially reconstruct the following frames. Moreover, we build an extended BIRNAT-color algorithm for color videos aiming at joint reconstruction and demosaicing. Extensive results on both video and spectral, simulation and real data from three SCI cameras demonstrate the superior performance of BIRNAT.

computer science, artificial intelligence,engineering, electrical & electronic
CoCoCs: Co-Optimized Compressive Imaging Driven by High-Level Vision

Honghao Huang,Chengyang Hu,Jingwei Li,Xiaowen Dong,Hongwei Chen

DOI: https://doi.org/10.1364/oe.468733

IF: 3.8

2022-01-01

Optics Express

Abstract:Compressive imaging senses optically encoded high-dimensional scene data with far fewer measurements and then performs reconstruction via appropriate algorithms. In this paper, we present a novel noniterative end-to-end deep learning-based framework for compressive imaging, dubbed CoCoCs. In comparison to existing approaches, we extend the pipeline by co-optimizing the recovery algorithm with optical coding as well as cascaded high-level computer vision tasks to boost the quality of the reconstruction. We demonstrate the proposed framework on two typical compressive imaging systems, i.e., single pixel imaging and snapshot video compressive imaging. Extensive results, including conventional image quality criteria, mean opinion scores, and accuracy in image classification and motion recognition, confirm that CoCoCs can yield realistic images and videos, which are friendly to both human viewing and computer vision. We hope CoCoCs will give impetus to bridge the gap between compressive imagers and computer vision and the perception of human.
Content Adaptive and Error Propagation Aware Deep Video Compression

Guo Lu,Chunlei Cai,Xiaoyun Zhang,Li Chen,Wanli Ouyang,Dong Xu,Zhiyong Gao

DOI: https://doi.org/10.1007/978-3-030-58536-5_27

2020-01-01

Abstract:Recently, learning based video compression methods attract increasing attention. However, the previous works suffer from error propagation due to the accumulation of reconstructed error in inter predictive coding. Meanwhile, the previous learning based video codecs are also not adaptive to different video contents. To address these two problems, we propose a content adaptive and error propagation aware video compression system. Specifically, our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Based on the learned long-term temporal information, our approach effectively alleviates error propagation in reconstructed frames. More importantly, instead of using the hand-crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system. The proposed approach updates the parameters for encoder according to the rate-distortion criterion but keeps the decoder unchanged in the inference stage. Therefore, the encoder is adaptive to different video contents and achieves better compression performance by reducing the domain gap between the training and testing datasets. Our method is simple yet effective and outperforms the state-of-the-art learning based video codecs on benchmark datasets without increasing the model size or decreasing the decoding speed.
Content-aware Facial Image Compression with Deep Learning Method

Shuzhan Hu,Yiping Duan,Xiaoming Tao,Yongjia Liu,Xuming Zhang,Jianhua Lu

DOI: https://doi.org/10.1109/wcsp49889.2020.9299680

2020-01-01

Abstract:Recently, online office work and education through video conferences have become a typical part of people's daily life, increasing the need for bandwidth and storage of wireless communication networks. To avoid video stalling and blurring, this paper proposes a content-aware facial image compression method based on deep learning networks at low bit rates for higher quality of experience (QoE). In this paper, the average face calculated from the database collected by the server is incorporated into the face representation model, which is used as prior knowledge to improve the performance of the video communication system. With the face representation model, facial images are decomposed into shape components and texture components, and different compression methods are applied to these components according to different redundancy levels. Specifically, a convolutional neural network (CNN) method is proposed to process texture components. The texture components are optimized by end-to-end rate-distortion learning with mean squared error (MSE) and structural similarity (SSIM) as optimization objectives for high fidelity reconstruction. The shape components use quantizer and entropy coding to preserve the details. Compared with traditional compression methods JPEG and JPEG2000, our proposed method could achieve a better performance in terms of PSNR and SSIM values at low bit rates.
Key Frames Assisted Hybrid Encoding for High-Quality Compressive Video Sensing

H. O. N. G. H. A. O. Huang,J. I. A. J. I. E. Teng,Y. U. Liang,C. H. E. N. G. Y. A. N. G. Hu,M. I. N. G. H. U. A. Chen,S. I. G. A. N. G. Yang,H. O. N. G. W. E. Chen

DOI: https://doi.org/10.1364/oe.471754

IF: 3.8

2022-01-01

Optics Express

Abstract:Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct high-quality video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.
DeepCoder: A Deep Neural Network Based Video Compression

Tong Chen,Haojie Liu,Qiu Shen,Tao Yue,Xun Cao,Zhan Ma

DOI: https://doi.org/10.1109/vcip.2017.8305033

2017-01-01

Abstract:Inspired by recent advances in deep learning, we present the DeepCoder — a Convolutional Neural Network (CNN) based video compression framework. We apply separate CNN nets for predictive and residual signals respectively. Scalar quantization and Huffman coding are employed to encode the quantized feature maps (fMaps) into binary stream. We use the fixed 32 × 32 block in this work to demonstrate our ideas, and performance comparison is conducted with the well-known H.264/AVC video coding standard with comparable rate-distortion performance. Here distortion is measured using Structural Similarity (SSIM) because it is more close to perceptual response.
Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging

Xin Yuan,Yang Liu,Jinli Suo,Qionghai Dai

DOI: https://doi.org/10.48550/arXiv.2003.13654

2020-07-18

Abstract:Snapshot compressive imaging (SCI) aims to capture the high-dimensional (usually 3D) images using a 2D sensor (detector) in a single snapshot. Though enjoying the advantages of low-bandwidth, low-power and low-cost, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging. The bottleneck lies in the reconstruction algorithms; they are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the widely used PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload and prove the convergence of PnP-GAP under the SCI hardware constraints. By employing deep denoising priors, we first time show that PnP can recover a UHD color video ($3840\times 1644\times 48$ with PNSR above 30dB) from a snapshot 2D measurement. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm. The code is available at <a class="link-external link-https" href="https://github.com/liuyang12/PnP-SCI" rel="external noopener nofollow">this https URL</a>.

Image and Video Processing,Computer Vision and Pattern Recognition
Plug-and-Play Algorithms for Video Snapshot Compressive Imaging

Xin Yuan,Yang Liu,Jinli Suo,Frédo Durand,Qionghai Dai,Fredo Durand

DOI: https://doi.org/10.1109/tpami.2021.3099035

IF: 23.6

2021-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:We consider the reconstruction problem of video snapshot compressive imaging (SCI), which captures high-speed videos using a low-speed 2D sensor (detector). The underlying principle of SCI is to modulate sequential high-speed frames with different masks and then these encoded frames are integrated into a snapshot on the sensor and thus the sensor can be of low-speed. On one hand, video SCI enjoys the advantages of low-bandwidth, low-power and low-cost. On the other hand, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging and one of the bottlenecks lies in the reconstruction algorithm. Existing algorithms are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload. We first employ the image deep denoising priors to show that PnP can recover a UHD color video with 30 frames from a snapshot measurement. Since videos have strong temporal correlation, by employing the video deep denoising priors, we achieve a significant improvement in the results. Furthermore, we extend the proposed PnP algorithms to the color SCI system using mosaic sensors, where each pixel only captures the red, green or blue channels. A joint reconstruction and demosaicing paradigm is developed for flexible and high quality reconstruction of color video SCI systems. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm.

computer science, artificial intelligence,engineering, electrical & electronic
Snapshot Interferometric 3D Imaging by Compressive Sensing and Deep Learning

Mu Qiao,Yangyang Sun,Jiawei Ma,Ziyi Meng,Xuan Liu,Xin Yuan

DOI: https://doi.org/10.48550/arXiv.2004.02633

2020-04-03

Image and Video Processing

Abstract:We demonstrate single-shot compressive three-dimensional (3D) $(x, y, z)$ imaging based on interference coding. The depth dimension of the object is encoded into the interferometric spectra of the light field, resulting a $(x, y, \lambda)$ datacube which is subsequently measured by a single-shot spectrometer. By implementing a compression ratio up to $400$, we are able to reconstruct $1G$ voxels from a 2D measurement. Both an optimization based compressive sensing algorithm and a deep learning network are developed for 3D reconstruction from a single 2D coded measurement. Due to the fast acquisition speed, our approach is able to capture volumetric activities at native camera frame rates, enabling 4D (volumetric-temporal) visualization of dynamic scenes.
Deep Image Compression via End-to-End Learning

Haojie Liu,Tong Chen,Qiu Shen,Tao Yue,Zhan Ma

DOI: https://doi.org/10.48550/arXiv.1806.01496

2018-06-05

Abstract:We present a lossy image compression method based on deep convolutional neural networks (CNNs), which outperforms the existing BPG, WebP, JPEG2000 and JPEG as measured via multi-scale structural similarity (MS-SSIM), at the same bit rate. Currently, most of the CNNs based approaches train the network using a L2 loss between the reconstructions and the ground-truths in the pixel domain, which leads to over-smoothing results and visual quality degradation especially at a very low bit rate. Therefore, we improve the subjective quality with the combination of a perception loss and an adversarial loss additionally. To achieve better rate-distortion optimization (RDO), we also introduce an easy-to-hard transfer learning when adding quantization error and rate constraint. Finally, we evaluate our method on public Kodak and the Test Dataset P/M released by the Computer Vision Lab of ETH Zurich, resulting in averaged 7.81% and 19.1% BD-rate reduction over BPG, respectively.

Image and Video Processing,Computer Vision and Pattern Recognition
Provable deep video denoiser using spatial-temporal information for video snapshot compressive imaging: Algorithm and convergence analysis

Baoshun Shi,Dan Li,Yuxin Wang,Yueming Su,Qiusheng Lian

DOI: https://doi.org/10.1016/j.sigpro.2023.109236

IF: 4.729

2023-09-02

Signal Processing

Abstract:Video snapshot compressive imaging (SCI) is a new compressive imaging system that aims to recover multiple video frames from a single measurement. Recent plug-and-play (PnP) imaging methods have been used for solving SCI inverse problems by leveraging pre-trained deep Gaussian denoisers. Under the condition of diminishing noise levels, a necessary assumption for the fixed-point convergence of PnP imaging algorithms is that the plugged denoisers are bounded denoisers. However, it is difficult to prove that existing deep Gaussian denoisers meet this assumption due to the complex network architectures, limiting the convergence analysis of these algorithms. This paper aims to elaborate a bounded deep video denoiser to remedy such a gap. Concretely, we propose a denoiser using double tight frames dubbed as VDTF and plug it into the PnP framework to construct the PnP-VDTF algorithm. In VDTF, a constant network (CNet) is designed, wherein a Swin Transformer-based module and a 3D convolution module extract the spatial information of the global features and the temporal information of the input video, respectively. Theoretically, we provide a strict boundary proof of VDTF, and show that PnP-VDTF can generate fixed-point convergent trajectories. Experiments show that PnP-VDTF can achieve higher-quality reconstructions compared with benchmark SCI algorithms.

engineering, electrical & electronic

Video Snapshot Compressive Imaging Via Deep Learning with Adversarial Training

Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement

BIRNAT: Bidirectional Recurrent Neural Networks with Adversarial Training for Video Snapshot Compressive Imaging

Deep Learning for Snapshot Image Compressive Sensing

Deep learning for video compressive sensing

Deep Optics for Video Snapshot Compressive Imaging

Hybrid CNN-Transformer Architecture for Efficient Large-Scale Video Snapshot Compressive Imaging

Key frames assisted hybrid encoding for photorealistic compressive video sensing

EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging

Compressed Video Sensing Based on Deep Generative Adversarial Network

Recurrent Neural Networks for Snapshot Compressive Imaging

CoCoCs: Co-Optimized Compressive Imaging Driven by High-Level Vision

Content Adaptive and Error Propagation Aware Deep Video Compression

Content-aware Facial Image Compression with Deep Learning Method

Key Frames Assisted Hybrid Encoding for High-Quality Compressive Video Sensing

DeepCoder: A Deep Neural Network Based Video Compression

Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imaging

Plug-and-Play Algorithms for Video Snapshot Compressive Imaging

Snapshot Interferometric 3D Imaging by Compressive Sensing and Deep Learning

Deep Image Compression via End-to-End Learning

Provable deep video denoiser using spatial-temporal information for video snapshot compressive imaging: Algorithm and convergence analysis