Abstract:Recently, deep learning (DL) presents a promising performance in the joint classification of multimodal remote sensing (RS) data. However, most of the approaches adopt a supervised learning (SL) manner, where the discrimination capability is limited by the paucity of labeled samples. Though some attempts have been made to develop semi-supervised methods, they prefer to select the highly confident predictions as pseudo-ground truth and discard those unreliable ones. Actually, unreliable samples can also provide useful information, e.g., indicating the categories to which samples may belong and definitely not belong. Focused on this, a novel uncertainty-aware contrastive learning (UACL) method is proposed. Here, label uncertainty analysis (UA) based on multilevel probability estimation is first conducted to separate reliable and unreliable samples, which are then processed with a designed hybrid ("hard" or "soft") contrastive learning (CL) strategy. For reliable samples, the "hard" CL pushes the network to learn features that will minimize the intra-class distance while maximizing the inter-class distance, according to the pseudo-labels. For unreliable samples, the "soft" CL aims to learn the similarity and difference among samples, where the predicted class probabilities are queried to estimate a soft mask for an adaptive feature similarity measurement. Moreover, a multimodal spectral-spatial joint feature representation pipeline of triple branches, i.e., one spectral branch for hyperspectral images (HSIs) and two spatial branches for multimodal data, is also introduced. By jointly learning from both labeled and unlabeled samples, more discriminative spectral-spatial feature representation will lead to a further boost in classification performance. Extensive experiments on four well-known multimodal datasets prove the effectiveness of the proposed semi-supervised classification method. Codes are available at https://github.com/Ding-Kexin/UACL.

Contrastive Learning for Urban Land Cover Classification With Multimodal Siamese Network

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

Cross-Modal Contrastive Learning for Remote Sensing Image Classification.

Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification

Coarse-to-Fine Contrastive Self-Supervised Feature Learning for Land-Cover Classification in SAR Images With Limited Labeled Data

Self-supervised SAR-optical Data Fusion and Land-cover Mapping using Sentinel-1/-2 Images

Multimodal Supervised Contrastive Learning in Remote Sensing Downstream Tasks

Unsupervised Representation High-Resolution Remote Sensing Image Scene Classification Via Contrastive Learning Convolutional Neural Network

Beyond pixels: Learning from multimodal hyperspectral superpixels for land cover classification

SiSL-Net: Saliency-guided self-supervised learning network for image classification

Contrastive Learning Based on Multiscale Hard Features for Remote-Sensing Image Scene Classification.

Assisted learning for land use classification: The important role of semantic correlation between heterogeneous images

Heterogeneous Network Based Contrastive Learning Method for PolSAR Land Cover Classification

Enhancing Hyperspectral Image Prediction with Contrastive Learning in Low-Label Regime

Uncertainty-Aware Contrastive Learning for Semi-Supervised Classification of Multimodal Remote Sensing Images

Multimodal Semantic Consistency-Based Fusion Architecture Search for Land Cover Classification

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Remote Sensing Collaborative Classification Using Multimodal Adaptive Modulation Network

SAR Image Classification Using Contrastive Learning and Pseudo-Labels with Limited Data

Multimodal Contrastive Learning for Remote Sensing Image Feature Extraction Based on Relaxed Positive Samples

Purified Contrastive Learning With Global and Local Representation for Hyperspectral Image Classification