Restoring Images in Adverse Weather Conditions via Histogram Transformer

Shangquan Sun,Wenqi Ren,Xinwei Gao,Rui Wang,Xiaochun Cao

2024-07-25

Abstract:Transformer-based image restoration methods in adverse weather have achieved significant progress. Most of them use self-attention along the channel dimension or within spatially fixed-range blocks to reduce computational load. However, such a compromise results in limitations in capturing long-range spatial features. Inspired by the observation that the weather-induced degradation factors mainly cause similar occlusion and brightness, in this work, we propose an efficient Histogram Transformer (Histoformer) for restoring images affected by adverse weather. It is powered by a mechanism dubbed histogram self-attention, which sorts and segments spatial features into intensity-based bins. Self-attention is then applied across bins or within each bin to selectively focus on spatial features of dynamic range and process similar degraded pixels of the long range together. To boost histogram self-attention, we present a dynamic-range convolution enabling conventional convolution to conduct operation over similar pixels rather than neighbor pixels. We also observe that the common pixel-wise losses neglect linear association and correlation between output and ground-truth. Thus, we propose to leverage the Pearson correlation coefficient as a loss function to enforce the recovered pixels following the identical order as ground-truth. Extensive experiments demonstrate the efficacy and superiority of our proposed method. We have released the codes in Github.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problem of significant degradation of image quality under adverse weather conditions (such as rain, fog, snow, etc.). Specifically, these weather conditions can severely affect the performance of computer vision tasks, such as object detection and depth estimation. Therefore, restoring images affected by adverse weather is crucial for visual aesthetics and safety.

### Main Problems and Solutions in the Paper

#### 1. **Limitations of Existing Methods**
Existing Transformer - based image restoration methods usually reduce the computational load in the following two ways:
- Use self - attention mechanisms along the channel dimension.
- Apply self - attention mechanisms within a fixed - range spatial block.

However, these compromises lead to limitations in the ability to capture long - distance spatial features, thus affecting the effect of image restoration.

#### 2. **Proposed New Method**
To solve the above problems, the author proposes a new model named **Histoformer**, with the core of **Histogram Transformer**. This model effectively deals with weather - induced image degradation problems by introducing the **Histogram Self - Attention (HSA)**.

##### Key Technical Points:
- **Dynamic - range Histogram Self - Attention (DHSA)**: Group spatial features by intensity and apply self - attention mechanisms between or within these groups to selectively focus on spatial features within the dynamic range and simultaneously handle pixels with similar long - distance degradations.

- **Dynamic - range Convolution**: Enable traditional convolutions to operate on similar pixels rather than being limited to neighboring pixels.

- **Pearson Correlation Coefficient Loss**: Ensure that the restored pixels maintain the same order relationship as the ground truth,弥补了传统像素级损失函数忽视线性关联和相关性的不足.

### Summary
The core problem of the paper is to improve the quality of image restoration under adverse weather conditions. To this end, the author proposes the Histoformer model. By introducing DHSA, dynamic - range convolution, and the Pearson Correlation Coefficient Loss, it effectively solves the limitations of existing methods in capturing long - distance spatial features, thereby achieving more efficient global degradation removal.

### Formula Display
To better understand the technical details in the paper, the following is the Markdown - format display of several key formulas:

- **Reconstruction Loss**:
\[
L_{\text{rec}}=\|I_{\text{hq}} - I_{\text{gt}}\|_1
\]

- **Pearson Correlation Coefficient**:
\[
\rho(I_{\text{hq}}, I_{\text{gt}})=\frac{\sum_{i = 1}^{3HW}(I_{\text{hq}_i}-\bar{I}_{\text{hq}})(I_{\text{gt}_i}-\bar{I}_{\text{gt}})}{3HW\sigma(I_{\text{hq}})\sigma(I_{\text{gt}})}
\]

- **Correlation Loss**:
\[
L_{\text{cor}}=\frac{1}{2}(1 - \rho(I_{\text{hq}}, I_{\text{gt}}))
\]

- **Overall Loss Function**:
\[
L = L_{\text{rec}}+ \alpha L_{\text{cor}}
\]

Through these improvements, Histoformer performs well in multiple...