Single-Image Deraining via Recurrent Residual Multiscale Networks

Yupei Zheng,Xin Yu,Miaomiao Liu,Shunli Zhang

DOI: https://doi.org/10.1109/tnnls.2020.3041752

IF: 14.255

2022-03-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Existing deraining approaches represent rain streaks with different rain layers and then separate the layers from the background image. However, because of the complexity of real-world rain, such as various densities, shapes, and directions of rain streaks, it is very difficult to decompose a rain image into clean background and rain layers. In this article, we develop a novel single-image deraining method based on residual multiscale pyramid to mitigate the difficulty of rain image decomposition. To be specific, we progressively remove rain streaks in a coarse-to-fine fashion, where heavy rain is first removed in coarse-resolution levels and then light rain is eliminated in fine-resolution levels. Furthermore, based on the observation that residuals between a restored image and its corresponding rain image give critical clues of rain streaks, we regard the residuals as an attention map to remove rains in the consecutive finer level image. To achieve a powerful yet compact deraining framework, we construct our network by recurrent layers and remove rain with the same network in different pyramid levels. In addition, we design a multiscale kernel selection network (MSKSN) to facilitate our single network to remove rain streaks at different levels. In this manner, we reduce 81% of the model parameters without decreasing deraining performance compared with our prior work. Extensive experimental results on widely used benchmarks show that our approach achieves superior deraining performance compared with the state of the art.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture

What problem does this paper attempt to address?

### The Problem Addressed by the Paper

This paper aims to solve the problem of single image rain removal. Specifically:

1. **Issues with Existing Methods**:
- Existing rain removal methods typically decompose rain streaks into different layers and separate these layers from the background image. However, due to the complexity of real-world rain (such as varying densities, shapes, and directions), it is challenging to decompose a rain image into a clean background and rain layers.

2. **Proposed New Method**:
- This paper proposes a new method based on a residual multi-scale pyramid to alleviate the difficulty of rain image decomposition. The method removes rain streaks step by step, first removing heavy rain at low-resolution layers and then removing light rain at high-resolution layers. Additionally, by observing the residuals between the restored image and its corresponding rain image, key clues of rain streaks can be obtained. Therefore, these residuals are treated as attention maps to remove continuous fine-grained levels of rain.

3. **Network Design and Optimization**:
- To build a robust and compact rain removal framework, the authors use recursive layers and employ the same network to remove rain at different pyramid levels. Furthermore, a multi-scale kernel selection network (MSKSN) is designed to facilitate the removal of rain streaks at different levels by a single network. This approach reduces the model parameters by 81% without compromising rain removal performance.

4. **Experimental Results**:
- Extensive experimental results demonstrate that the proposed method outperforms existing techniques in rain removal performance on commonly used benchmarks.

In summary, this paper addresses the problem of single image rain removal by proposing a novel method based on a residual multi-scale pyramid, achieving significant results in practical applications.