Abstract:Recently, deep learning (DL) presents a promising performance in the joint classification of multimodal remote sensing (RS) data. However, most of the approaches adopt a supervised learning (SL) manner, where the discrimination capability is limited by the paucity of labeled samples. Though some attempts have been made to develop semi-supervised methods, they prefer to select the highly confident predictions as pseudo-ground truth and discard those unreliable ones. Actually, unreliable samples can also provide useful information, e.g., indicating the categories to which samples may belong and definitely not belong. Focused on this, a novel uncertainty-aware contrastive learning (UACL) method is proposed. Here, label uncertainty analysis (UA) based on multilevel probability estimation is first conducted to separate reliable and unreliable samples, which are then processed with a designed hybrid ("hard" or "soft") contrastive learning (CL) strategy. For reliable samples, the "hard" CL pushes the network to learn features that will minimize the intra-class distance while maximizing the inter-class distance, according to the pseudo-labels. For unreliable samples, the "soft" CL aims to learn the similarity and difference among samples, where the predicted class probabilities are queried to estimate a soft mask for an adaptive feature similarity measurement. Moreover, a multimodal spectral-spatial joint feature representation pipeline of triple branches, i.e., one spectral branch for hyperspectral images (HSIs) and two spatial branches for multimodal data, is also introduced. By jointly learning from both labeled and unlabeled samples, more discriminative spectral-spatial feature representation will lead to a further boost in classification performance. Extensive experiments on four well-known multimodal datasets prove the effectiveness of the proposed semi-supervised classification method. Codes are available at https://github.com/Ding-Kexin/UACL.

Multimodal Deep Learning for Semisupervised Classification of Hyperspectral and LiDAR Data

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Dual-Stream Class-Adaptive Network for Semi-Supervised Hyperspectral Image Classification

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data

Superpixel-Based Long-Range Dependent Network for High-Resolution Remote-Sensing Image Classification

Shared-Private Decoupling-Based Multilevel Feature Alignment Semisupervised Learning for HSI and LiDAR Classification

Local Weight Coupled Network: Multi-Modal Unequal Semi-Supervised Domain Adaptation.

Learning transferable cross-modality representations for few-shot hyperspectral and LiDAR collaborative classification

Dual-Branch Subpixel-Guided Network for Hyperspectral Image Classification

Self-Supervised Learning With Multiscale Densely Connected Network for Hyperspectral Image Classification

Crossmodal Sequential Interaction Network for Hyperspectral and LiDAR Data Joint Classification

Uncertainty-Aware Contrastive Learning for Semi-Supervised Classification of Multimodal Remote Sensing Images

Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques

A Unified Multimodal Deep Learning Framework for Remote Sensing Imagery Classification.

Dual-input ultralight multi-head self-attention learning network for hyperspectral image classification

Dual-Branch Feature Fusion Network Based Cross-Modal Enhanced CNN and Transformer for Hyperspectral and LiDAR Classification

Multimodal Attention-Aware Convolutional Neural Networks for Classification of Hyperspectral and LiDAR Data

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

Semisupervised deep learning using consistency regularization and pseudolabels for hyperspectral image classification