Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining

Xiang Chen,Jinshan Pan,Jiangxin Dong

2024-04-02

Abstract:How to effectively explore multi-scale representations of rain streaks is important for image deraining. In contrast to existing Transformer-based methods that depend mostly on single-scale rain appearance, we develop an end-to-end multi-scale Transformer that leverages the potentially useful features in various scales to facilitate high-quality image reconstruction. To better explore the common degradation representations from spatially-varying rain streaks, we incorporate intra-scale implicit neural representations based on pixel coordinates with the degraded inputs in a closed-loop design, enabling the learned features to facilitate rain removal and improve the robustness of the model in complex scenarios. To ensure richer collaborative representation from different scales, we embed a simple yet effective inter-scale bidirectional feedback operation into our multi-scale Transformer by performing coarse-to-fine and fine-to-coarse information communication. Extensive experiments demonstrate that our approach, named as NeRD-Rain, performs favorably against the state-of-the-art ones on both synthetic and real-world benchmark datasets. The source code and trained models are available at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily addresses the issue of image deraining and proposes a novel method aimed at improving the quality of derained images through multi-scale feature representation.

The core contributions of the paper include:

1. **Multi-Scale Transformer**: An effective multi-scale Transformer is developed, capable of generating high-quality deraining results and utilizing multi-scale rain information for exploration and utilization.

2. **Implicit Neural Representation (INR)**: The introduction of implicit neural representation technology is used to better learn complex rain degradation features, demonstrating that this method aids in rain removal and enhances the model's robustness in complex scenarios.

3. **Bidirectional Feedback Mechanism**: A simple yet effective bidirectional feedback propagation operation is integrated into the proposed multi-scale Transformer to promote better feature interaction between different scales.

Specifically, to handle raindrop effects in images, the researchers designed a bidirectional multi-scale Transformer architecture, which includes two key components: a branch for extracting scale-specific features (i.e., the Transformer branch) and a branch for learning common rain degradation representations (the INR branch). By combining these components, the method can effectively model and remove raindrops at different scales, particularly excelling in handling complex and randomly distributed raindrops.

Experimental results show that the proposed method (named NeRD-Rain) outperforms the state-of-the-art techniques on both synthetic and real-world datasets, especially when facing diverse raindrop shapes in practical applications. Additionally, the method demonstrates good model efficiency, with lower floating-point operations (FLOPs) and fewer training parameters.

In summary, by introducing innovative technologies such as multi-scale analysis, implicit neural representation, and bidirectional feedback mechanisms, this paper provides an effective solution to the image deraining problem, significantly improving image quality and visual effects.