Deep Learning for Snapshot Image Compressive Sensing

Haomiao Zhang,Ping Wang,Xin Yuan

DOI: https://doi.org/10.1364/3d.2023.jw2a.11

2023-01-01

Abstract:This work proposes a novel image compression approach by spatially multiplexing the non-local information of image, which is a potential implementation of snapshot image compressive sensing.

What problem does this paper attempt to address?

Content-Adaptive Image Compressed Sensing Using Deep Learning

Liqun Zhong,Shuai Wan,Leyi Xie,Shun Zhang

DOI: https://doi.org/10.23919/apsipa.2018.8659768

2018-01-01

Abstract:This paper proposes a framework of content-adaptive image compressed sensing using deep learning, which analyzes the image content and adaptively allocates samples for different image patches accordingly. Experimental results demonstrate that the proposed framework outperforms the state-of-the-arts both in subjective and objective quality, especially at low sampling rates. For example, when the sampling rate is 0.1, 1-6 dB improvement in peak signal to noise ratio (PSNR) is observed. Moreover, the proposed work reconstructs images with more details and less image blocking effects, leading to apparent visual improvement.
Adaptive Sampling for Image Compressed Sensing Based on Deep Learning

Liqun Zhong,Shuai Wan,Leyi Xie

DOI: https://doi.org/10.1088/1742-6596/1229/1/012016

2019-01-01

Abstract:The compressed sensing (CS) theory has been applied to image compression successfully as most image signals are sparse in a certain domain. In this paper, we focus on how to improve the sampling efficiency for network-based image compressed sensing by using our proposed adaptive sampling algorithm. We conduct content adaptive sampling to achieve a significant improvement. Experiments results indicate that our proposed framework outperforms the state-of-the-arts both in subjective and objective quality. An average of 1-6 dB improvement in peak signal to noise ratio (PSNR) is observed. Moreover, the proposed work reconstructs images with more details and less image blocking effects, leading to apparent visual improvement.
Deep-Learning Supervised Snapshot Compressive Imaging Enabled by an End-to-End Adaptive Neural Network.

Miguel Marquez,Yingming Lai,Xianglei Liu,Cheng Jiang,Shian Zhang,Henry Arguello,Jinyang Liang

DOI: https://doi.org/10.1109/jstsp.2022.3172592

IF: 7.695

2022-01-01

IEEE Journal of Selected Topics in Signal Processing

Abstract:Snapshot compressive imaging (SCI) is an advanced approach for single-shot high-dimensional data visualization. Deep learning is popularly used to improve SCI's performance. However, most existing methods are merely used as a replacement for analytical-modeling-based image reconstruction. Moreover, these models cling to the conventional random coded apertures and often presume a linear shearing operation. To overcome these limitations, we develop a new end-to-end convolutional neural network, termed deep high-dimensional adaptive net (D-HAN) that offers multi-faceted supervision to SCI by optimizing the coded aperture, sensing the shearing operation, and reconstructing three-dimensional datacubes. The D-HAN is implemented in two representative SCI systems for ultrahigh-speed imaging and hyperspectral imaging. The D-HAN is envisioned to benefit SCI in system design, image reconstruction, and performance evaluation.
Snapshot Interferometric 3D Imaging by Compressive Sensing and Deep Learning

Mu Qiao,Yangyang Sun,Jiawei Ma,Ziyi Meng,Xuan Liu,Xin Yuan

DOI: https://doi.org/10.48550/arXiv.2004.02633

2020-04-03

Image and Video Processing

Abstract:We demonstrate single-shot compressive three-dimensional (3D) $(x, y, z)$ imaging based on interference coding. The depth dimension of the object is encoded into the interferometric spectra of the light field, resulting a $(x, y, \lambda)$ datacube which is subsequently measured by a single-shot spectrometer. By implementing a compression ratio up to $400$, we are able to reconstruct $1G$ voxels from a 2D measurement. Both an optimization based compressive sensing algorithm and a deep learning network are developed for 3D reconstruction from a single 2D coded measurement. Due to the fast acquisition speed, our approach is able to capture volumetric activities at native camera frame rates, enabling 4D (volumetric-temporal) visualization of dynamic scenes.
Snapshot Multiplexed Imaging Based on Compressive Sensing.

Ying Fu,Chen Sun,Lizhi Wang,Hua Huang

DOI: https://doi.org/10.1007/978-3-030-00767-6_43

2018-01-01

Abstract:Multiplexed imaging methods have been proposed to extend the field of view (FoV) of the imaging devices. However, the nature of multiple exposures hinders its application in time-crucial scenarios. In this paper, we design a snapshot multiplexed imaging system for wide FoV imaging. In the system, the scene is first spatially encoded by a mask, and then the coded scene is optically divided into multiple sub-regions which are finally superimposed and measured on a sensor array. We model the demultiplexing as a compressive sensing (CS) reconstruction problem and introduce two methods, one is based on Total Variation (TV) constraint and the other is based on sparsity constraint, to reconstruct the scene. Simulation results demonstrate the effectiveness of the proposed system.
Recent Advances of Deep Learning for Spectral Snapshot Compressive Imaging

Jinli Suo,Zongliang Wu,Zhangyuan Li,Xin Yuan

DOI: https://doi.org/10.1117/12.2641748

2022-01-01

Abstract:We review the recent deep learning reconstruction algorithms for spectral snapshot compressive imaging (SCI), which used a single shot measurement to capture the three-dimensional (3D, x, y, lambda) spectral image. Recent years, deep learning has been the dominant algorithm to conduct reconstruction due to high speed and accuracy. Various frameworks such as end-to-end neural networks, deep unfolding, plug-and-play networks have been developed. Furthermore, the untrained neural networks have also been used. In this paper, we review diverse deep learning methods for spectral SCI. In addition to the aforementioned frameworks, different backbones and network structures including the most recent Transformers are reviewed. Simulation and real data results are presented to compare these methods.
Video Snapshot Compressive Imaging Via Deep Learning with Adversarial Training

Honghao Huang,Chengyang Hu,Sigang Yang,Minghua Chen,Hongwei Chen

DOI: https://doi.org/10.1364/cleo_at.2022.jw3b.172

2022-01-01

Abstract:We present deep learning with adversarial training for video compressive imaging, which can recover 50 frames from a snapshot with boosted visual quality compared to typical iterative algorithm and neural network with simple loss function.
Hyperspectral Snapshot Compressive Imaging with Non-Local Spatial-Spectral Residual Network

Ying Yang,Yong Xie,Xunhao Chen,Yubao Sun

DOI: https://doi.org/10.3390/rs13091812

IF: 5

2021-01-01

Remote Sensing

Abstract:Snapshot Compressive Imaging is an emerging technology that is based on compressive sensing theory to achieve high-efficiency hyperspectral data acquisition. The core problem of this technology is how to reconstruct 3D hyperspectral data from the 2D snapshot measurement in a fast and high-quality manner. In this paper, we propose a novel deep network, which consists of the symmetric residual module and the non-local spatial-spectral attention module, to learn the reconstruction mapping in a data-driven way. The symmetric residual module uses symmetric residual connections to improve the potential of interaction between convolution operations and further promotes the fusion of local features. The non-local spatial-spectral attention module is designed to capture the non-local spatial-spectral correlation in the hyperspectral image. Specifically, this module calculates the channel attention matrix to capture the global correlations between all of the spectral channels, and it fuses the channel attention attained feature maps and the spatial attention weighted features as the module output, thus both of the spatial-spectral correlations of hyperspectral images can be fully utilized for reconstruction. In addition, a compound loss, including the reconstruction loss, the measurement loss, and the cosine loss, is designed to guide the end-to-end network learning. We experimentally evaluate the proposed method on simulation and real datasets. The experimental results show that the proposed network outperforms the competing methods in terms of the reconstruction quality and running time.
Deep learning for video compressive sensing

Mu Qiao,Ziyi Meng,Jiawei Ma,Xin Yuan

DOI: https://doi.org/10.1063/1.5140721

IF: 5.6

2020-03-01

APL Photonics

Abstract:We investigate deep learning for video compressive sensing within the scope of snapshot compressive imaging (SCI). In video SCI, multiple high-speed frames are modulated by different coding patterns and then a low-speed detector captures the integration of these modulated frames. In this manner, each captured measurement frame incorporates the information of all the coded frames, and reconstruction algorithms are then employed to recover the high-speed video. In this paper, we build a video SCI system using a digital micromirror device and develop both an end-to-end convolutional neural network (E2E-CNN) and a Plug-and-Play (PnP) framework with deep denoising priors to solve the inverse problem. We compare them with the iterative baseline algorithm GAP-TV and the state-of-the-art DeSCI on real data. Given a determined setup, a well-trained E2E-CNN can provide video-rate high-quality reconstruction. The PnP deep denoising method can generate decent results without task-specific pre-training and is faster than conventional iterative algorithms. Considering speed, accuracy, and flexibility, the PnP deep denoising method may serve as a baseline in video SCI reconstruction. To conduct quantitative analysis on these reconstruction algorithms, we further perform a simulation comparison on synthetic data. We hope that this study contributes to the applications of SCI cameras in our daily life.

optics,physics, applied
Deep Learning for Visual Data Compression

Guo Lu,Ren Yang,Shenlong Wang,Shan Liu,Radu Timofte

DOI: https://doi.org/10.1145/3474085.3478877

2021-01-01

Abstract:In this paper, we will introduce the recent progress in deep learning based visual data compression, including image compression, video compression and point cloud compression. In the past few years, deep learning techniques have been successfully applied to various computer vision and image processing applications. However, for the data compression task, the traditional approaches (i.e., block based motion estimation and motion compensation, etc.) are still widely employed in the mainstream codecs. Considering the powerful representation capability of neural networks, it is feasible to improve the data compression performance by employing the advanced deep learning technologies. To this end, the deep leaning based compression approaches have recently received increasing attention from both academia and industry in the field of computer vision and signal processing.
Snapshot compressed sensing: performance bounds and algorithms

Shirin Jalali,Xin Yuan

DOI: https://doi.org/10.48550/arXiv.1808.03661

2019-04-30

Abstract:Snapshot compressed sensing (CS) refers to compressive imaging systems in which multiple frames are mapped into a single measurement frame. Each pixel in the acquired frame is a noisy linear mapping of the corresponding pixels in the frames that are combined together. While the problem can be cast as a CS problem, due to the very special structure of the sensing matrix, standard CS theory cannot be employed to study such systems. In this paper, a compression-based framework is employed for theoretical analysis of snapshot CS systems. It is shown that this framework leads to two novel, computationally-efficient and theoretically-analyzable compression-based recovery algorithms. The proposed methods are iterative and employ compression codes to define and impose the structure of the desired signal. Theoretical convergence guarantees are derived for both algorithms. In the simulations, it is shown that, in the cases of both noise-free and noisy measurements, combining the proposed algorithms with a customized video compression code, designed to exploit nonlocal structures of video frames, significantly improves the state-of-the-art performance.

Information Theory
Snapshot Compressive Imaging: Principle, Implementation, Theory, Algorithms and Applications

Xin Yuan,David J. Brady,Aggelos K. Katsaggelos

DOI: https://doi.org/10.1109/MSP.2020.3023869

2021-03-08

Abstract:Capturing high-dimensional (HD) data is a long-term challenge in signal processing and related fields. Snapshot compressive imaging (SCI) uses a two-dimensional (2D) detector to capture HD ($\ge3$D) data in a {\em snapshot} measurement. Via novel optical designs, the 2D detector samples the HD data in a {\em compressive} manner; following this, algorithms are employed to reconstruct the desired HD data-cube. SCI has been used in hyperspectral imaging, video, holography, tomography, focal depth imaging, polarization imaging, microscopy, \etc.~Though the hardware has been investigated for more than a decade, the theoretical guarantees have only recently been derived. Inspired by deep learning, various deep neural networks have also been developed to reconstruct the HD data-cube in spectral SCI and video SCI. This article reviews recent advances in SCI hardware, theory and algorithms, including both optimization-based and deep-learning-based algorithms. Diverse applications and the outlook of SCI are also discussed.

Image and Video Processing,Computer Vision and Pattern Recognition
End-to-end Snapshot Compressed Super-Resolution Imaging with Deep Optics

Bo Zhang,Xin Yuan,Chao Deng,Zhihong Zhang,Jinli Suo,Qionghai Dai

DOI: https://doi.org/10.1364/optica.450657

IF: 10.4

2022-01-01

Optica

Abstract:Capturing both fine structure and high dynamics of the scene is demanded in many applications. However, such high throughput recording requires significant transmission bandwidth and large storage. Off-the-shelf super-resolution and temporal compressive sensing can partially address this challenge, but directly concatenating the two techniques fails to boost throughput because of artifact accumulation and magnification in sequential reconstruction. In this Letter, we propose an encoded capturing approach to simultaneously increase spatial and temporal resolvability with a low-bandwidth camera sensor. Specifically, we introduce point-spread-function (PSF) engineering via deep optics to encode fine spatial details and temporal compressive sensing to encode fast motions into a low-resolution snapshot. Furthermore, we develop an end-to-end deep neural network to optimize PSF and retrieve high-throughput videos from a compactly compressed measurement. Trained on simulation data and fine-tuned to fit system settings, our end-to-end system offers a 128 × data throughput compared to conventional imaging.
Spatial-Temporal Synergic Prior Driven Unfolding Network for Snapshot Compressive Imaging

Zhuoyuan Wu,Zhenyu Zhang,Jiechong Sing,Man Zhang

DOI: https://doi.org/10.1109/ICME51207.2021.9428320

2021-01-01

Abstract:In order to develop a fast and accurate algorithm for snapshot compressive sensing (SCI), we combine the merits of two kinds of existing SCI methods: the interpretability of traditional model-based methods and the speed of learning-based ones. Concretely, we build a novel Spatial-Temporal synErgic Prior driven unfolding network for SCI, dubbed STEP-SCI, which is inspired by Half-Quadratic Splittin...
Deep Blind Compressed Sensing

Shikha Singh,Vanika Singhal,Angshul Majumdar

DOI: https://doi.org/10.48550/arXiv.1612.07453

2016-12-22

Abstract:This work addresses the problem of extracting deeply learned features directly from compressive measurements. There has been no work in this area. Existing deep learning tools only give good results when applied on the full signal, that too usually after preprocessing. These techniques require the signal to be reconstructed first. In this work we show that by learning directly from the compressed domain, considerably better results can be obtained. This work extends the recently proposed framework of deep matrix factorization in combination with blind compressed sensing; hence the term deep blind compressed sensing. Simulation experiments have been carried out on imaging via single pixel camera, under-sampled biomedical signals, arising in wireless body area network and compressive hyperspectral imaging. In all cases, the superiority of our proposed deep blind compressed sensing can be envisaged.

Computer Vision and Pattern Recognition
Deep-learning-based Single-Photon-counting Compressive Imaging Via Jointly Trained Subpixel Convolution Sampling.

Wen-Cheng Li,Qiu-Rong Yan,Yan-Qiu Guan,Sheng-Tao Yang,Cong Peng,Zhe-Yu Fang

DOI: https://doi.org/10.1364/ao.394410

IF: 1.9

2020-01-01

Applied Optics

Abstract:The combination of single-pixel-imaging and single-photon-counting technology can achieve ultrahigh-sensitivity photon-counting imaging. However, its applications in high-resolution and real-time scenarios are limited by the long sampling and reconstruction time. Deep-learning-based compressive sensing provides an effective solution due to its ability to achieve fast and high-quality reconstruction. This paper proposes a sampling and reconstruction integrated neural network for single-photon-counting compressive imaging. To effectively remove the blocking artefact, a subpixel convolutional layer is jointly trained with a deep reconstruction network to imitate compressed sampling. By modifying the forward and backward propagation of the network, the first layer is trained into a binary matrix, which can be applied to the imaging system. An improved deep-reconstruction network based on the traditional Inception network is proposed, and the experimental results show that its reconstruction quality is better than existing deep-learning-based compressive sensing reconstruction algorithms.
Combining Low-Rank and Deep Plug-and-Play Priors for Snapshot Compressive Imaging

Yong Chen,Xinfeng Gui,Jinshan Zeng,Xi-Le Zhao,Wei He

DOI: https://doi.org/10.1109/tnnls.2023.3294262

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Snapshot compressive imaging (SCI) is a promising technique that captures a 3-D hyperspectral image (HSI) by a 2-D detector in a compressed manner. The ill-posed inverse process of reconstructing the HSI from their corresponding 2-D measurements is challenging. However, current approaches either neglect the underlying characteristics, such as high spectral correlation, or demand abundant training datasets, resulting in an inadequate balance among performance, generalizability, and interpretability. To address these challenges, in this article, we propose a novel approach called LR2DP that integrates the model-driven low-rank prior and data-driven deep priors for SCI reconstruction. This approach not only captures the spectral correlation and deep spatial features of HSI but also takes advantage of both model-based and learning-based methods without requiring any extra training datasets. Specifically, to preserve the strong spectral correlation of the HSI effectively, we propose that the HSI lies in a low-rank subspace, thereby transforming the problem of reconstructing the HSI into estimating the spectral basis and spatial representation coefficient. Inspired by the mutual promotion of unsupervised deep image prior (DIP) and trained deep denoising prior (DDP), we integrate the unsupervised network and pre-trained deep denoiser into the plug-and-play (PnP) regime to estimate the representation coefficient together, aiming to explore the internal target image prior (learned by DIP) and the external training image prior (depicted by pre-trained DDP) of the HSI. An effective half-quadratic splitting (HQS) technique is employed to optimize the proposed HSI reconstruction model. Extensive experiments on both simulated and real datasets demonstrate the superiority of the proposed method over the state-of-the-art approaches.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing

Yan Yang,Jian Sun,Huibin Li,Zongben Xu

DOI: https://doi.org/10.1109/tpami.2018.2883941

IF: 23.6

2020-03-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Compressive sensing (CS) is an effective technique for reconstructing image from a small amount of sampled data. It has been widely applied in medical imaging, remote sensing, image compression, etc. In this paper, we propose two versions of a novel deep learning architecture, dubbed as ADMM-CSNet, by combining the traditional model-based CS method and data-driven deep learning method for image reconstruction from sparsely sampled measurements. We first consider a generalized CS model for image reconstruction with undetermined regularizations in undetermined transform domains, and then two efficient solvers using Alternating Direction Method of Multipliers (ADMM) algorithm for optimizing the model are proposed. We further unroll and generalize the ADMM algorithm to be two deep architectures, in which all parameters of the CS model and the ADMM algorithm are discriminatively learned by end-to-end training. For both applications of fast CS complex-valued MR imaging and CS imaging of real-valued natural images, the proposed ADMM-CSNet achieved favorable reconstruction accuracy in fast computational speed compared with the traditional and the other deep learning methods.

computer science, artificial intelligence,engineering, electrical & electronic
Deep Image Compression with Residual Learning

Weigui Li,Wenyu Sun,Yadong Zhao,Zhuqing Yuan,Yongpan Liu

DOI: https://doi.org/10.3390/app10114023

2020-01-01

Abstract:An end-to-end image compression framework based on deep residual learning is proposed. Three levels of residual learning are adopted to improve the compression quality: (1) the ResNet structure; (2) the deep channel residual learning for quantization; and (3) the global residual learning in full resolution. Residual distribution is commonly a single Gaussian distribution, and relatively easy to be learned by the neural network. Furthermore, an attention model is combined in the proposed framework to compress regions of an image with different bits adaptively. Across the experimental results on Kodak PhotoCD test set, the proposed approach outperforms JPEG and JPEG2000 by PSNR and MS-SSIM at low BPP (bit per pixel). Furthermore, it can produce much better visual quality. Compared to the state-of-the-art deep learning-based codecs, the proposed approach also achieves competitive performance.
An End-to-End Deep Learning Image Compression Framework Based on Semantic Analysis

Cheng Wang,Yifei Han,Weidong Wang

DOI: https://doi.org/10.3390/app9173580

2019-09-01

Applied Sciences

Abstract:Lossy image compression can reduce the bandwidth required for image transmission in a network and the storage space of a device, which is of great value in improving network efficiency. With the rapid development of deep learning theory, neural networks have achieved great success in image processing. In this paper, inspired by the diverse extent of attention in human eyes to each region of the image, we propose an image compression framework based on semantic analysis, which creatively combines the application of deep learning in the field of image classification and image compression. We first use a convolutional neural network (CNN) to semantically analyze the image, obtain the semantic importance map, and propose a compression bit allocation algorithm to allow the recurrent neural network (RNN)-based compression network to hierarchically compress the image according to the semantic importance map. Experimental results validate that the proposed compression framework has better visual quality compared with other methods at the same compression ratio.

Deep Learning for Snapshot Image Compressive Sensing

Content-Adaptive Image Compressed Sensing Using Deep Learning

Adaptive Sampling for Image Compressed Sensing Based on Deep Learning

Deep-Learning Supervised Snapshot Compressive Imaging Enabled by an End-to-End Adaptive Neural Network.

Snapshot Interferometric 3D Imaging by Compressive Sensing and Deep Learning

Snapshot Multiplexed Imaging Based on Compressive Sensing.

Recent Advances of Deep Learning for Spectral Snapshot Compressive Imaging

Video Snapshot Compressive Imaging Via Deep Learning with Adversarial Training

Hyperspectral Snapshot Compressive Imaging with Non-Local Spatial-Spectral Residual Network

Deep learning for video compressive sensing

Deep Learning for Visual Data Compression

Snapshot compressed sensing: performance bounds and algorithms

Snapshot Compressive Imaging: Principle, Implementation, Theory, Algorithms and Applications

End-to-end Snapshot Compressed Super-Resolution Imaging with Deep Optics

Spatial-Temporal Synergic Prior Driven Unfolding Network for Snapshot Compressive Imaging

Deep Blind Compressed Sensing

Deep-learning-based Single-Photon-counting Compressive Imaging Via Jointly Trained Subpixel Convolution Sampling.

Combining Low-Rank and Deep Plug-and-Play Priors for Snapshot Compressive Imaging

ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing

Deep Image Compression with Residual Learning

An End-to-End Deep Learning Image Compression Framework Based on Semantic Analysis