Abstract:Data-driven single image deraining (SID) models have achieved greater progress by simulations, but there is still a large gap between current deraining performance and practical high-level applications, since high-level semantic information is usually neglected in current studies. Although few studies jointly considered high-level tasks (e.g., segmentation) to enable the model to learn more high-level information, there are two obvious shortcomings. First, they require the segmentation labels for training, limiting their operations on other datasets without high-level labels. Second, high- and low-level information are not fully interacted, hence having limited improvement in both deraining and segmentation tasks. In this paper, we propose a Semantic Guided Interactive Network (SGINet), which considers the sufficient interaction between SID and semantic segmentation using a three-stage deraining manner, i.e., coarse deraining, semantic information extraction, and semantics guided deraining. Specifically, a Full Resolution Module (FRM) without down-/up-sampling is proposed to predict the coarse deraining images without context damage. Then, a Segmentation Extracting Module (SEM) is designed to extract accurate semantic information. We also develop a novel contrastive semantic discovery (CSD) loss, which can instruct the process of semantic segmentation without real semantic segmentation labels. Finally, a triple-direction U-net-based Semantic Interaction Module (SIM) takes advantage of the coarse deraining images and semantic information for fully interacting low-level with high-level tasks. Extensive simulations on the newly-constructed complex datasets Cityscapes_syn and Cityscapes_real demonstrated that our model could obtain more promising results. Overall, our SGINet achieved SOTA deraining and segmentation performance in both simulation and real-scenario data, compared with other representative SID methods.

ESDINet: Efficient Shallow-Deep Interaction Network for Semantic Segmentation of High-Resolution Aerial Images

EHANet: Efficient Hybrid Attention Network Towards Real-time Semantic Segmentation

ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

ISDNet: Integrating Shallow and Deep Networks for Efficient Ultra-high Resolution Segmentation

DESENet: a bilateral network with detail-enhanced semantic encoder for real-time semantic segmentation

ELANet: an efficiently lightweight asymmetrical network for real-time semantic segmentation

EARMNet:A Fast and Accurate Semantic Segmentation Network with Lightweight Efficient Asymmetric Residual Module

Semantic segmentation for remote sensing images via dense feature extraction and companion loss neural network

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation

SGINet: Toward Sufficient Interaction Between Single Image Deraining and Semantic Segmentation

Senet: Spatial Information Enhancement for Semantic Segmentation Neural Networks

SPNet: Dual-Branch Network with Spatial Supplementary Information for Building and Water Segmentation of Remote Sensing Images

Edge-guided Nonlinear Dynamic Convolution Network for Lightweight Semantic Segmentation

Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network

LMANet: A Lightweight Asymmetric Semantic Segmentation Network Based on Multi-Scale Feature Extraction

DSANet: Dilated Spatial Attention for Real-Time Semantic Segmentation in Urban Street Scenes.

Dsnet: Accelerate Indoor Scene Semantic Segmentation

FastICENet: A Real-Time and Accurate Semantic Segmentation Model for Aerial Remote Sensing River Ice Image

Efficient Multi-scale Network for Semantic Segmentation of fine-Resolution Remotely Sensed Images

Light-Deeplabv3+: a lightweight real-time semantic segmentation method for complex environment perception