An Effective Image Restorer: Denoising and Luminance Adjustment for Low-photon-count Imaging

Shansi Zhang,Edmund Y. Lam

DOI: https://doi.org/10.48550/arXiv.2110.15715

2021-11-02

Abstract:Imaging under photon-scarce situations introduces challenges to many applications as the captured images are with low signal-to-noise ratio and poor luminance. In this paper, we investigate the raw image restoration under low-photon-count conditions by simulating the imaging of quanta image sensor (QIS). We develop a lightweight framework, which consists of a multi-level pyramid denoising network (MPDNet) and a luminance adjustment (LA) module to achieve separate denoising and luminance enhancement. The main component of our framework is the multi-skip attention residual block (MARB), which integrates multi-scale feature fusion and attention mechanism for better feature representation. Our MPDNet adopts the idea of Laplacian pyramid to learn the small-scale noise map and larger-scale high-frequency details at different levels, and feature extractions are conducted on the multi-scale input images to encode richer contextual information. Our LA module enhances the luminance of the denoised image by estimating its illumination, which can better avoid color distortion. Extensive experimental results have demonstrated that our image restorer can achieve superior performance on the degraded images with various photon levels by suppressing noise and recovering luminance and color effectively.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problem of image restoration under low - photon - number imaging conditions. Specifically, when the number of photons is very small, the captured images usually have a low signal - to - noise ratio and poor brightness, which poses a challenge to image restoration. To address this problem, the author has developed a lightweight framework, which consists of a multi - level pyramid denoising network (MPDNet) and a brightness adjustment (LA) module, used for noise removal and brightness enhancement respectively.

### Main Problems and Solutions

1. **Image Denoising under Low - Photon - Number Imaging**
- **Problem**: Under low - photon - number conditions, the image is severely affected by noise, with a low signal - to - noise ratio and is difficult to restore.
- **Solution**: The author proposes a multi - level pyramid denoising network (MPDNet), using the idea of Laplacian pyramid to learn small - scale noise maps and larger - scale high - frequency details at different scales. Through multi - scale input image feature extraction, more abundant context information is encoded, thus effectively removing noise.

2. **Brightness Enhancement**
- **Problem**: The denoised image may have insufficient brightness and needs further enhancement.
- **Solution**: The author designs a brightness adjustment (LA) module, which enhances the brightness by estimating the illumination of the input image while avoiding color distortion. The LA module adopts an encoder - decoder architecture, combining convolutional layers and multi - scale feature fusion to preserve the color information of the input image.

### Technical Contributions

1. **Multi - Hop Attention Residual Block (MARB)**
- Integrates multi - scale feature fusion and attention mechanism to obtain better feature representation.
- Has far fewer parameters than the common residual block (RB), improving the efficiency of the model.

2. **Multi - Level Pyramid Denoising Network (MPDNet)**
- Adopts the idea of Laplacian pyramid to learn noise maps and high - frequency details at different scales.
- Introduces multi - scale input image feature extraction to encode more abundant context information.

3. **Brightness Adjustment Module (LA)**
- Enhances the brightness of the denoised image and makes adaptive adjustments according to the estimated illumination.
- Fully preserves the color information of the input image and avoids color distortion.

### Experimental Results

- **Data Sets**: Experiments are carried out using the Pascal VOC2012, COCO val2017 and Kodak data sets.
- **Performance Evaluation**: The performance of the model is evaluated through indicators such as PSNR and SSIM. The experimental results show that this framework performs excellently when processing low - photon - number images and can effectively remove noise and restore brightness and color.

### Conclusion

This paper proposes an effective image restoration method specifically for image denoising and brightness enhancement under low - photon - number imaging conditions. Through the multi - level pyramid denoising network and the brightness adjustment module, this method can effectively restore image quality at various photon levels and has high practical value.