Rainy day image semantic segmentation based on two-stage progressive network

Heng Zhang,Dongli Jia,Hui Ma

DOI: https://doi.org/10.1007/s00371-024-03287-5

IF: 2.835

2024-02-26

The Visual Computer

Abstract:Semantic segmentation plays a crucial role in the fields of computer vision and computer graphics, with extensive applications in various practical scenarios. Significant progress has been made in semantic segmentation tasks using deep learning-based methods. However, most existing semantic segmentation algorithms focus on good weather conditions, and they face challenges in terms of accuracy and robustness when applied to rainy scenes due to factors such as raindrops, haze, and lighting variations. To address this issue, this paper proposes a rainy-day semantic segmentation method based on a two-stage progressive network. The proposed method consists of two modules: a rain removal module responsible for eliminating raindrops and haze from the input rainy images and restoring the basic structural information of the images, and a segmentation module that performs pixel-level semantic prediction on the rain-removed images. Specifically, the rain removal module introduces two progressive units with shared weights to gradually achieve rain removal. The segmentation module adopts an encoder–decoder architecture, utilizing down-sampling and deep asynchronous bottleneck units for encoding. It also introduces a dual attention-guided fusion module to aggregate channel attention information and spatial attention information, guiding the multiscale feature fusion process in the decoder. Experimental results demonstrate that this method effectively mitigates the influence of rain streaks on semantic segmentation, thereby improving segmentation performance and achieving more accurate and robust semantic segmentation results in rainy conditions. We will provide the code and datasets on https://github.com/zhang152267/TSPN.

computer science, software engineering

What problem does this paper attempt to address?

The paper proposes a new solution to the problem of image semantic segmentation in rainy environments. Specifically, existing semantic segmentation algorithms perform well under good weather conditions, but face challenges in accuracy and robustness in rainy scenes due to factors such as raindrops, haze, and lighting changes. To address this issue, the authors propose a rainy image semantic segmentation method based on a two-stage progressive network.

This method mainly includes two modules:
1. **Rain Removal Module**: Responsible for eliminating raindrops and haze in the input rainy image and restoring the basic structural information of the image.
2. **Segmentation Module**: Performs pixel-level semantic prediction on the image after rain removal processing.

The rain removal module employs two progressive units with shared weights to gradually remove raindrops. The segmentation module adopts an encoder-decoder architecture and uses downsampling and Depthwise Asynchronous Bottleneck Units (DAB Units) for encoding. Additionally, this module introduces a Dual Attention Guided Fusion Module (DAGFM) to aggregate channel attention information and spatial attention information, guiding the multi-scale feature fusion process in the decoder.

Experimental results show that this method can effectively mitigate the impact of rain streaks on semantic segmentation, thereby improving segmentation performance and achieving more accurate and robust semantic segmentation results under rainy conditions.