Abstract:Currently, the mainstream restoration tasks under adverse weather conditions have predominantly focused on single-weather scenarios. However, in reality, multiple weather conditions always coexist and their degree of mixing is usually unknown. Under such complex and diverse weather conditions, single-weather restoration models struggle to meet practical demands. This is particularly critical in fields such as autonomous driving, where there is an urgent need for a model capable of effectively handling mixed weather conditions and enhancing image quality in an automated manner. In this paper, we propose a Task Sequence Generator module that, in conjunction with the Task Intra-patch Block, effectively extracts task-specific features embedded in degraded images. The Task Intra-patch Block introduces an external learnable sequence that aids the network in capturing task-specific information. Additionally, we employ a histogram-based transformer module as the backbone of our network, enabling the capture of both global and local dynamic range features. Our proposed model achieves state-of-the-art performance on public datasets.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
The paper aims to solve the problem of image restoration under complex and changeable weather conditions. Currently, the mainstream image restoration tasks mainly focus on a single - weather scenario. However, in practical applications, multiple weather conditions often co - exist simultaneously, and the degree of their mixture is usually unknown. Such complex and diverse weather conditions make it difficult for a single - weather restoration model to meet the actual needs. Especially in fields such as autonomous driving, a model that can effectively handle mixed - weather conditions and automatically enhance image quality is required.
Specifically, the paper points out:
1. **Limitations of existing methods**: Most of the existing image restoration methods mainly target a single - weather condition, such as rain, snow or fog, etc., and cannot effectively handle the complex situation where multiple weather conditions co - exist simultaneously.
2. **Actual requirements**: In application scenarios such as autonomous driving, a model that can handle multiple weather conditions is required to improve the reliability and accuracy of the computer vision system.
3. **Technical challenges**: How to extract task - specific features under complex multi - weather conditions and effectively restore image quality.
### Solutions
To solve the above problems, the paper proposes a new multi - weather image restoration method, which mainly includes the following parts:
1. **Task Sequence Generator module**: This module, used in combination with the Task Intra - patch Block, can effectively extract task - specific features embedded in the degraded image.
2. **Task Intra - patch Block**: An external learnable sequence is introduced to help the network capture task - specific information.
3. **Histogram - based Transformer module**: As the backbone of the network, it can capture global and local dynamic - range features.
### Main contributions
- **Proposing a novel and efficient solution**: By using weather degradation information and task feature sequence generation, the problem of image restoration in bad weather is effectively solved. The experimental results show that this method outperforms the existing state - of - the - art methods in real - data sets and downstream object detection tasks.
- **Task Sequence Generator**: A new feature extraction block is designed, which can effectively capture the detailed features of multiple degradation types and generate rich stage - specific features, applicable to each stage of the image restoration process.
- **Histogram - based Transformer block**: Provides a spatial attention mechanism, focuses on the degradation dynamic range caused by the weather, and achieves globally effective degradation removal.
### Formula summary
1. **Fog formation model**:
\[
I(x)=J(x)t(x)+A(1 - t(x))
\]
where \(I(x)\) is the radiance of the degraded foggy image, \(J(x)\) is the fog - free target scene, \(A\) is the global atmospheric light, and \(t(x)\) is the medium transmission map, which depends on the unknown depth information.
2. **Rain streak degradation model**:
\[
I = J+\sum_{i}S_{i}
\]
where \(I\) is the observed input image, \(J\) is the rain - free background scene, \(S_{i}\) is the rain layer, and \(n\) is the total number of rain streaks.
3. **Snow scene degradation model**:
\[
I(x)=K(x)T(x)+A(x)(1 - T(x))
\]
where \(I(x)\) represents the snow scene image, \(T(x)\) and \(A(x)\) represent the transmission map and atmospheric light respectively, and \(K(x)\) is the unobstructed snow scene, which can be decomposed into \(K(x)=J(x)(1 - Z(x)R(x))+C(x)Z(x)R(x)\), where \(J(x)\) is the clean image, \(R(x)\) is a binary mask representing the position information of snow, and \(Z(x)\) and \(C(x)\) represent the color difference image and snow mask respectively.
4. **Task Intra - patch Block output**:
\[
TIPB_{i}(