Abstract:Data-driven single image deraining (SID) models have achieved greater progress by simulations, but there is still a large gap between current deraining performance and practical high-level applications, since high-level semantic information is usually neglected in current studies. Although few studies jointly considered high-level tasks (e.g., segmentation) to enable the model to learn more high-level information, there are two obvious shortcomings. First, they require the segmentation labels for training, limiting their operations on other datasets without high-level labels. Second, high- and low-level information are not fully interacted, hence having limited improvement in both deraining and segmentation tasks. In this paper, we propose a Semantic Guided Interactive Network (SGINet), which considers the sufficient interaction between SID and semantic segmentation using a three-stage deraining manner, i.e., coarse deraining, semantic information extraction, and semantics guided deraining. Specifically, a Full Resolution Module (FRM) without down-/up-sampling is proposed to predict the coarse deraining images without context damage. Then, a Segmentation Extracting Module (SEM) is designed to extract accurate semantic information. We also develop a novel contrastive semantic discovery (CSD) loss, which can instruct the process of semantic segmentation without real semantic segmentation labels. Finally, a triple-direction U-net-based Semantic Interaction Module (SIM) takes advantage of the coarse deraining images and semantic information for fully interacting low-level with high-level tasks. Extensive simulations on the newly-constructed complex datasets Cityscapes_syn and Cityscapes_real demonstrated that our model could obtain more promising results. Overall, our SGINet achieved SOTA deraining and segmentation performance in both simulation and real-scenario data, compared with other representative SID methods.

Interaction semantic segmentation network via progressive supervised learning

SGINet: Toward Sufficient Interaction Between Single Image Deraining and Semantic Segmentation

SPNet: Dual-Branch Network with Spatial Supplementary Information for Building and Water Segmentation of Remote Sensing Images

Lightweight and Progressively-Scalable Networks for Semantic Segmentation

ESDINet: Efficient Shallow-Deep Interaction Network for Semantic Segmentation of High-Resolution Aerial Images

Inter-Level Feature Balanced Fusion Network for Street Scene Segmentation

RELAXNet: Residual Efficient Learning and Attention Expected Fusion Network for Real-Time Semantic Segmentation

Semantics Recalibration and Detail Enhancement Network for Real‐time Semantic Segmentation

Semantic Segmentation of Remote-Sensing Images Based on Multiscale Feature Fusion and Attention Refinement

HSNet: an Intelligent Hierarchical Semantic-Aware Network System for Real-Time Semantic Segmentation

Senet: Spatial Information Enhancement for Semantic Segmentation Neural Networks

Semantic segmentation for remote sensing images via dense feature extraction and companion loss neural network

A Spectral–Spatial Context-Boosted Network for Semantic Segmentation of Remote Sensing Images

RSI-Net: Two-Stream Deep Neural Network for Remote Sensing Images-Based Semantic Segmentation

MiSSNet: Memory-Inspired Semantic Segmentation Augmentation Network for Class-Incremental Learning in Remote Sensing Images

Stair Fusion Network With Context-Refined Attention for Remote Sensing Image Semantic Segmentation

Learning a Discriminative Feature Network for Semantic Segmentation

ASFNet: Adaptive Multiscale Segmentation Fusion Network for Real‐time Semantic Segmentation

Semantic Segmentation Network of Remote Sensing Images with Dynamic Loss Fusion Strategy.

Cross-scale Graph Interaction Network for Semantic Segmentation of Remote Sensing Images

Point Cloud Semantic Segmentation with Cross-Correction Features