Abstract:Cloud detection is a crucial step in remote sensing image preprocessing. Deep learning (DL) methods are currently preferred for cloud detection, and the original images need to be cropped into smaller patches for DL training and testing. However, in very-high-resolution (VHR) remote sensing cloud-snow coexistence scenes, the cropped patches have low feature richness and high inter-class feature similarity. As a result, the common DL methods are prone to misclassifying clouds in complex regions. Therefore, based on the expanded cloud detection dataset of VHR cloud-snow coexistence regions, this paper proposes a multi-task driven and reconfigurable network (MTDR-Net) to better perform the cloud detection work in the scenes. There are four modules in MTDR-Net: the high-resolution backbone with multiple projection heads module (HRB-MPH), the re-parametrizable multi-scale feature fusion module (RP-MFF), the lightweight and adaptive feature fusion module (LAFF), and the multi-task gradient flow guidance module (MTGFG). HRB-MPH facilitates global information interaction, enhances the multi-level granularity feature representation, and supports feature sharing between pixel-level and superpixel-level segmentation tasks. RP-MFF is used to capture the local multi-scale cloud features in the training stage and simplify losslessly the structure in the testing stage. LAFF is employed to reconstruct meaningful cloud features. MTGFG can provide unbiased guidance on dividing gradient flow between the multi-tasks. The experimental results show that the MTDR-Net obtains the greatest accuracy performance due to its anti-interference capacity for confusing ground objects and its excellent cloud boundary and thin cloud extraction ability. In addition, MTDR-Net has the fewest parameters, forward-reasoning floating point operations, memory footprint, and memory access volume. With the coarse-to-fine strategy, MTDR-Net can be further sped up by avoiding inference resource consumption in cloud-free/all-cloud scenes.

Cloud Type Classification Using Multi-modal Information Based on Multi-task Learning

Cloud Recognition Using Multimodal Information: A Review

Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network

Multi-View Ground-Based Cloud Recognition by Transferring Deep Visual Information

Multi-modal microblog classification via multi-task learning

Multimodal Ground-Based Remote Sensing Cloud Classification Via Learning Heterogeneous Deep Features.

Improving Cloud Type Classification of Ground-Based Images Using Region Covariance Descriptors

A Multi-Task Framework for Weather Recognition

Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks.

Cross-Domain Ground-Based Cloud Classification Based on Transfer of Local Features and Discriminative Metric Learning

MT-HCCAR: Multi-Task Deep Learning with Hierarchical Classification and Attention-based Regression for Cloud Property Retrieval

Cloud type classification using deep learning with cloud images

Multi-modal Scene Categorization Using Multi-Tasks Learning

Deep Tensor Fusion Network for Multimodal Ground-Based Cloud Classification in Weather Station Networks

CloudY-Net: A Deep Convolutional Neural Network Architecture for Joint Segmentation and Classification of Ground-Based Cloud Images

Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

CMPN: A Multimodal Predictive Model for Cloud Trajectory Forecast

Multilevel Cloud Detection in Remote Sensing Images Based on Deep Learning

A multi-task driven and reconfigurable network for cloud detection in cloud-snow coexistence regions from very-high-resolution remote sensing images

Multi-Task Label Embedding for Text Classification