Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks

Zhilu Zhang,Shuohao Zhang,Renlong Wu,Zifei Yan,Wangmeng Zuo

2024-05-31

Abstract:It is highly desired but challenging to acquire high-quality photos with clear content in low-light environments. Although multi-image processing methods (using burst, dual-exposure, or multi-exposure images) have made significant progress in addressing this issue, they typically focus on specific restoration or enhancement problems, and do not fully explore the potential of utilizing multiple images. Motivated by the fact that multi-exposure images are complementary in denoising, deblurring, high dynamic range imaging, and super-resolution, we propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks in this work. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data and then adapts it to real-world unlabeled images. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed. Moreover, we construct a data simulation pipeline to synthesize pairs and collect real-world images from 200 nighttime scenarios. Experiments on both datasets show that our method performs favorably against the state-of-the-art multi-image processing ones. The dataset, code, and pre-trained models are available at <a class="link-external link-https" href="https://github.com/cszhilu1998/BracketIRE" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to obtain high - quality and clearly - contented photos in low - light environments. Specifically, the authors focus on how to unify image restoration and enhancement tasks through multi - image processing methods (such as using burst, double - exposure or multi - exposure images) to generate noise - free, blur - free, high - dynamic - range and high - resolution images in low - light conditions.

### Problem Background
When taking photos in low - light environments, the camera's exposure time setting faces a dilemma:
- **Low exposure time**: The number of captured photons is small, resulting in inevitable noise and invisible dark areas.
- **High exposure time**: It is easy to produce blurring caused by hand - shaking or object movement, and the bright areas may be over - exposed.

Although single - image restoration (such as denoising, deblurring, super - resolution) and enhancement (such as high - dynamic - range reconstruction) methods have been widely studied, their performance is limited when dealing with severely degraded problems. In addition, existing multi - image processing methods usually focus on specific restoration or enhancement problems and fail to fully utilize the potential of multiple images.

### Research Motivation
The authors point out that multi - exposure images are complementary in denoising, deblurring, high - dynamic - range imaging and super - resolution. Therefore, they propose to use exposure bracketing photography to unify these tasks. However, due to the difficulty in collecting real - world paired data, the authors suggest first pre - training with synthetic paired data and then applying it to real - world unlabeled images through a self - supervised adaptation method.

### Main Contributions
1. **Unifying Tasks**: Proposed to use exposure bracketing photography to unify image restoration and enhancement tasks, including image denoising, deblurring, high - dynamic - range reconstruction and super - resolution.
2. **Solution**: Proposed a solution to pre - train the model on synthetic paired data first and then apply it to real - world images through a self - supervised adaptation method. For this purpose, the Temporal Modulation Recurrent Network (TMRNet) and the self - supervised adaptation method were designed.
3. **Experimental Verification**: Experiments on synthetic and real - world datasets show that the proposed method outperforms existing multi - image processing methods.

### Method Overview
- **Problem Definition and Formulation**: Defined the specific form of the task by modeling the processing process of multi - exposure images.
- **Temporal Modulation Recurrent Network (TMRNet)**: In order to solve the degradation differences between different exposure images, a network structure that shares some parameters and assigns specific parameters to each frame was designed.
- **Self - supervised Real - image Adaptation**: By designing a self - supervised loss function, the model can better adapt to multi - exposure images in the real world.

### Summary
This paper aims to solve the problem of high - quality image generation in low - light environments through multi - exposure image processing techniques and proposes a new framework to unify multiple image restoration and enhancement tasks.