Ruoxi Zhu,Zhengzhong Tu,Jiaming Liu,Alan C. Bovik,Yibo Fan
Abstract:Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: <a class="link-external link-https" href="https://github.com/taco-group/MWFormer" rel="external noopener nofollow">this https URL</a>
What problem does this paper attempt to address?
This paper attempts to solve the problem of image degradation under various severe weather conditions. Specifically, most of the existing weather restoration methods can only handle specific types of degradation (such as rain, fog or snow), which is often insufficient in the real world, because multiple weather conditions (such as rain and snow or rain and fog) may occur simultaneously in actual scenes. The image degradation caused by these multiple weather conditions is more complex, and it is difficult for existing methods to deal with it effectively.
To solve this problem, the paper proposes a multi - weather Transformer model named MWFormer. MWFormer aims to solve the problem of image degradation caused by multiple weather conditions through a single unified architecture. The following are the main contributions of the paper:
1. **Introduced a new Transformer - based architecture**: MWFormer can use the same set of learning parameters to restore images degraded by multiple severe weather conditions.
2. **Adopted a hyper - network to extract content - independent, weather - aware features**: These features are used to dynamically modify the parameters of the restoration backbone network, thereby achieving adaptive restoration for different degradations and supporting other related applications.
3. **Feature vectors guide the behavior of the backbone network**: The feature vectors generated from the hyper - network can guide the behavior of the backbone network in all dimensions and scales (i.e., local space, global space and channel modulation).
4. **Created two variants**: One is used to reduce the computational cost, and the other is used to handle the mixed severe weather degradation not seen during training.
5. **Experimental results show superiority**: The experimental results on multiple benchmark datasets show that MWFormer significantly outperforms the existing state - of - the - art models in terms of visual and quantitative indicators. In addition, this method can be integrated into other network architectures to improve their performance in multi - weather restoration tasks.
### Formula Summary
- **Contrastive loss function**:
\[
L_{\text{con}}=\sum_{(a, b) \in P}\left[I(a, b)[m - d(v_a, v_b)]_++[1 - I(a, b)]d(v_a, v_b)\right]
\]
where \(P\) represents all possible image pairs in the batch, \(d(\cdot)\) represents cosine similarity, \(m\) is the positive margin value, and \(I(a, b)\) is an indicator indicating whether the two images contain the same weather degradation.
- **Feature extraction network output**:
\[
v = F_{\text{feat}}(I; \tau)
\]
- **Image restoration network output**:
\[
Y = F_{\text{res}}(I; \theta_{\text{fix}}, \theta_{\text{adap}}(v))
\]
- **Local - space adaptation**:
\[
W_{DWC}=\text{Reshape}(\text{Proj}(v))
\]
\[
\text{FFN}(X_{sl}, v)=\text{MLP}(\sigma(W_{DWC}*X_{sl}))
\]
- **Global - space adaptation**:
\[
W_i=\text{Reshape}(\text{Proj}(v)),\quad i = q, k, v
\]
\[
Q = X_{sg}W_q,\quad K = X_{sg}W_k,\quad V = X_{sg}W_v
\]
\[
\text{MSA}(X_{sg})=\text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V
\]
- **Channel - feature modulation**:
\[
X'=\gamma X+\beta
\]
Through these designs, MWFormer can not only handle multiple weather degradations in a single model, but also provides higher flexibility and controllability, which is suitable for different application scenarios.