Abstract:Recently, deep learning (DL) presents a promising performance in the joint classification of multimodal remote sensing (RS) data. However, most of the approaches adopt a supervised learning (SL) manner, where the discrimination capability is limited by the paucity of labeled samples. Though some attempts have been made to develop semi-supervised methods, they prefer to select the highly confident predictions as pseudo-ground truth and discard those unreliable ones. Actually, unreliable samples can also provide useful information, e.g., indicating the categories to which samples may belong and definitely not belong. Focused on this, a novel uncertainty-aware contrastive learning (UACL) method is proposed. Here, label uncertainty analysis (UA) based on multilevel probability estimation is first conducted to separate reliable and unreliable samples, which are then processed with a designed hybrid ("hard" or "soft") contrastive learning (CL) strategy. For reliable samples, the "hard" CL pushes the network to learn features that will minimize the intra-class distance while maximizing the inter-class distance, according to the pseudo-labels. For unreliable samples, the "soft" CL aims to learn the similarity and difference among samples, where the predicted class probabilities are queried to estimate a soft mask for an adaptive feature similarity measurement. Moreover, a multimodal spectral-spatial joint feature representation pipeline of triple branches, i.e., one spectral branch for hyperspectral images (HSIs) and two spatial branches for multimodal data, is also introduced. By jointly learning from both labeled and unlabeled samples, more discriminative spectral-spatial feature representation will lead to a further boost in classification performance. Extensive experiments on four well-known multimodal datasets prove the effectiveness of the proposed semi-supervised classification method. Codes are available at https://github.com/Ding-Kexin/UACL.

Triplet Contrastive Learning Framework With Adversarial Hard-Negative Sample Generation for Multimodal Remote Sensing Images

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

Uncertainty-Aware Contrastive Learning for Semi-Supervised Classification of Multimodal Remote Sensing Images

Adversarial Complementary Learning for Multisource Remote Sensing Classification

Multimodal Contrastive Learning for Remote Sensing Image Feature Extraction Based on Relaxed Positive Samples

A Novel Contrastive Learning Model for Aerial Images

ContrastNet: Unsupervised Feature Learning by Autoencoder and Prototypical Contrastive Learning for Hyperspectral Imagery Classification.

Transformer-based Contrastive Prototypical Clustering for Multimodal Remote Sensing Data.

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Spatial and Semantic Consistency Contrastive Learning for Self-Supervised Semantic Segmentation of Remote Sensing Images

Vision-Language Pre-Training with Triple Contrastive Learning

Multimodal Contrastive Learning for Unpaired and Depth-privileged Semantic Segmentation.

Contrastive Tokens and Label Activation for Remote Sensing Weakly Supervised Semantic Segmentation

Cross-Modal Contrastive Learning for Remote Sensing Image Classification.

P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding

Contrastive Multimodal Fusion with TupleInfoNCE

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

Modeling Inner- and Cross-Task Contrastive Relations for Continual Image Classification

Triplet Adversarial Domain Adaptation for Pixel-Level Classification of VHR Remote Sensing Images

Attention-Based Contrastive Learning for Few-Shot Remote Sensing Image Classification

Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification