Abstract:Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reconstruct a high - quality image sequence from 1 - bit quantum image data captured by a single - photon avalanche diode (SPAD) array under extremely low - light conditions while maintaining the original temporal and spatial resolutions. Traditional processing methods improve the interpretability of images through resampling in the spatio - temporal domain, but this will lead to the loss of temporal and spatial resolutions. Therefore, the paper proposes a new self - supervised method - bit2bit, aiming to directly reconstruct a high - quality image sequence from sparse binary quantum image data without sacrificing temporal and spatial resolutions. ### Specific Problem Description 1. **High - spatio - temporal - resolution image reconstruction**: How to reconstruct a high - quality image sequence with the original spatio - temporal resolution from 1 - bit binary image data captured by SPAD sensors? 2. **Noise handling**: How to effectively handle shot noise in binary images, especially when photons are very sparse? 3. **Self - supervised learning**: How to design a self - supervised learning method so that the model can be effectively trained without real labels? 4. **Over - fitting problem**: How to avoid over - fitting during the training process, especially when the data is sparse and noisy? ### Main Contributions 1. **Self - supervised denoising method**: Developed a new self - supervised method. By introducing a masking strategy to handle the characteristics of binary data, it effectively solves the shortcomings of traditional methods on binary data. 2. **Spatio - temporal information utilization**: Significantly improves the reconstruction quality by using spatio - temporal information. 3. **Network architecture and regularization techniques**: Explores different network architectures and regularization techniques to manage over - fitting and improve the reconstruction quality. 4. **New data set**: Provides a new data set containing real and simulated 1 - bit SPAD data to support further research and quantitative evaluation. ### Method Overview - **Data generation model**: Assume that each detection event is independent and follows a truncated Poisson distribution. In actual data processing, the SPAD sensor captures the result of a Bernoulli process. - **Data splitting**: Generate input and target pairs for training the self - supervised model by randomly splitting the original data. - **Masked loss function**: Introduce a masking strategy to exclude existing photon detection events in the input image to avoid artifacts during the training process. - **Network architecture**: Use a 3D ResUNet architecture, which can perform effective feature extraction and reconstruction in the spatio - temporal domain. ### Experimental Results - **Quantitative evaluation**: Experiments were carried out on simulated data and real data, and relatively high PSNR and SSIM values were obtained. - **Qualitative evaluation**: Demonstrates the quality of the reconstructed images. Especially under extremely low - light conditions, it can still maintain a good visual effect. In conclusion, through proposing a new self - supervised method, this paper successfully solves the challenge of reconstructing a high - quality image sequence from sparse binary quantum image data, providing a new solution for high - speed imaging under low - light conditions.

bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction

Quanta burst photography

Photon Counting Correction Method to Improve the Quality of Reconstructed Images in Single Photon Compressive Imaging Systems.

A Picture is Worth a Billion Bits: Real-Time Image Reconstruction from Dense Binary Pixels

Count-Free Single-Photon 3D Imaging with Race Logic

Super Sub-Nyquist Single-Pixel Imaging by Total Variation Ascending Ordering of the Hadamard Basis

Panoramas from Photons

Quanta Video Restoration

A SPAD-Based QVGA Image Sensor for Single-Photon Counting and Quanta Imaging

High Bit-Depth Image Acquisition Framework Using Embedded Quantization Bias

Number-State Reconstruction with a Single Single-Photon Avalanche Detector

Bit-plane Processing Techniques for Low-Light, High Speed Imaging with a SPAD-based QIS

Dynamic Low-light Imaging with Quanta Image Sensors

End-to-end Learned, Optically Coded Super-resolution SPAD Camera

Streaming quanta sensors for online, high-performance imaging and vision

Quantum imaging with a photon counting camera

High-resolution single-photon imaging with physics-informed deep learning

Single-Photon 3D Imaging with Equi-Depth Photon Histograms

A Plug-and-Play Algorithm for 3D Video Super-Resolution of Single-Photon LiDAR data

Adaptive compressed 3D imaging based on wavelet trees and Hadamard multiplexing with a single photon counting detector

Single photon imaging based on a photon driven sparse sampling