Abstract:Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the performance limitations of traditional cameras in low - light and high - speed imaging. Specifically: 1. **Imaging quality under low - light conditions**: - Traditional cameras are difficult to obtain high - quality images under low - light conditions because long - exposure will lead to motion blur, while short - exposure will produce high noise (Poisson noise). 2. **High - bandwidth requirements**: - High - speed imaging requires a high frame rate, which results in a huge amount of data and extremely high bandwidth requirements, making it difficult to implement in practical applications. 3. **Trade - off between low - light and high - speed imaging**: - There is a trade - off between low - light and high - speed imaging in traditional cameras, and it is impossible to meet the needs of these two aspects simultaneously. ### Solutions To solve the above problems, the paper proposes a method of combining event cameras and single - photon avalanche diode (SPADs) sensors. These two sensors have complementary advantages: - **SPADs**: It can achieve single - photon sensitivity under low - light conditions and has a microsecond - level time resolution. - **Event cameras**: It can measure luminance changes at a frequency of up to 1 MHz and has a low - bandwidth requirement. By fusing the data of these two sensors, the paper proposes a new sensor - fusion framework, which can perform low - light and high - speed imaging while reducing the bandwidth requirement. Specific methods include: 1. **Non - linear event - double - integration (NEDI)**: - Use the data of event cameras to de - blur SPAD images, overcoming the motion - blur problem of SPAD images caused by long - exposure. 2. **Asynchronous Kalman filter (AKF)**: - Combine the uncertainty models of SPAD and event cameras, and use the Kalman filter for asynchronous data fusion to generate HDR images in continuous time. 3. **Adaptive sampling**: - Utilize the uncertainty estimation of the Kalman filter to adaptively trigger the acquisition of SPAD frames, further reducing the bandwidth requirement. ### Experimental results The paper verifies the effectiveness of this method through simulation and real - data sets, and the results show that: - Under low - light conditions (> 10 Lux), this method can achieve high - speed imaging at 10 kHz. - Compared with the traditional frame - event - fusion method, this method reduces the bandwidth requirement by 4 times while maintaining the same imaging quality. - Compared with the method using only SPADs, this method improves the PSNR by 5 dB while reducing the bandwidth requirement. ### Summary By combining the advantages of SPADs and event cameras, this paper proposes a new sensor - fusion framework, which effectively solves the performance and bandwidth problems in low - light and high - speed imaging, and provides new technical means for fields such as robotics and medical imaging.

Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

End-to-end Learned, Optically Coded Super-resolution SPAD Camera

Event-Based Processing of Single Photon Avalanche Diode Sensors

Quanta burst photography

Generalized Event Cameras

Transforming Single Photon Camera Images to Color High Dynamic Range Images

Scintillation event imaging with a single photon avalanche diode camera

Smart-aggregation imaging for single molecule localization with SPAD cameras

Time-gated SPAD camera with reconfigurable macropixels for LIDAR applications

Event fields: Capturing light fields at high speed, resolution, and dynamic range

Low-latency automotive vision with event cameras

A Review of Image Reconstruction Based on Event Cameras

A megapixel time-gated SPAD image sensor for 2D and 3D imaging applications

Asynchronous Single-Photon 3D Imaging

Super-Resolution Time-Resolved Imaging using Computational Sensor Fusion

Quantized Fourier Ptychography with Binary Images from SPAD Cameras

A Plug-and-Play Algorithm for 3D Video Super-Resolution of Single-Photon LiDAR data

Simulating single-photon detector array sensors for depth imaging

Single-photon avalanche diode imagers in biophotonics: review and outlook

A Full Parallel Event Driven Readout Technique for Area Array SPAD FLIM Image Sensors.

FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D Imaging