Abstract:Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: <a class="link-external link-https" href="https://github.com/taco-group/MWFormer" rel="external noopener nofollow">this https URL</a>

UTDM: a universal transformer-based diffusion model for multi-weather-degraded images restoration

Vision Transformers for Single Image Dehazing

Multi-Weather Degradation-Aware Transformer for Image Restoration

MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers

Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations

ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions

Joint multi-dimensional dynamic attention and transformer for general image restoration

Universal Image Restoration with Text Prompt Diffusion

WeaFU: Weather-Informed Image Blind Restoration Via Multi-Weather Distribution Diffusion

Prompt-guided and degradation prior supervised transformer for adverse weather image restoration

Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration

Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration

DR-DiT: Image Deraining Using Diffusion Model with Transformer

Coarse-to-fine Mechanisms Mitigate Diffusion Limitations on Image Restoration

When Fast Fourier Transform Meets Transformer for Image Restoration

Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal

A Unified Conditional Framework for Diffusion-based Image Restoration

WaveDM: Wavelet-Based Diffusion Models for Image Restoration

Frequency-oriented Efficient Transformer for All-in-one Weather-degraded Image Restoration

Sparse Sampling Transformer with Uncertainty-Driven Ranking for Unified Removal of Raindrops and Rain Streaks