Abstract:The emerging research line of cross-modal learning focuses on the issue of transferring feature representation manner learned from limited multimodal data with labelings to the testing phase with partial modalities. This is essentially common and practical in the remote sensing (RS) community when only modal-incomplete data are in users' hands due to inevitable imaging or access restrictions under large-scale observation scenarios. However, most of the existing cross-modal learning methods have been designed with exclusive reliance on labeling, which can be either limited or noisy due to their costly production. To address this issue, we explore, in this article, the possibility to learn cross-modal feature representation in an unsupervised fashion. By integrating the multimodal data into a fully recombined matrix form, we propose the following: 1) the use of common subspace representation as the regression target instead of conventionally adopted binary labels and 2) the orthogonality and manifold alignment regularization terms to shrink the solution space while preserving the pairwise manifold correlations. Through this manner, the modality-specific and mutual latent representations in this common subspace as well as their corresponding projections can be learned simultaneously, and their optimums can be efficiently reached through a nearly one-step computation with the help of eigendecomposition. Finally, we show the superiority of our method through extensive image classification experiments on three multimodal datasets with four remotely sensed modalities involved (i.e., hyperspectral (HS), multispectral, synthetic aperture radar, and light detection and ranging (LiDAR) data). The code and dataset will be made freely available at https://github.com/jingyao16/UCSL to encourage the reproduction of our method and further use.

Beyond pixels: Learning from multimodal hyperspectral superpixels for land cover classification

CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences

SPLC: An Attention-Based End-to-End Superpixels and Land Cover Classification Jointly Learning Architecture for HR Remote Sensing Image

Emmcnn: An Etps-Based Multi-Scale And Multi-Feature Method Using Cnn For High Spatial Resolution Image Land-Cover Classification

A Multi-Hyperspectral Image Collaborative Mapping Model Based on Adaptive Learning for Fine Classification

Learning rich multimodal representation for robust land cover classification in fog

Land Cover Classification Based on the PSPNet and Superpixel Segmentation Methods with High Spatial Resolution Multispectral Remote Sensing Imagery

Efficient superpixel-oriented multi-task joint sparse representation classification for hyperspectral imagery

Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model

UCSL: Toward Unsupervised Common Subspace Learning for Cross-Modal Image Classification

Spectral-spatial based super pixel remote sensing image classification

Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

PSASL: Pixel-Level and Superpixel-Level Aware Subspace Learning for Hyperspectral Image Classification

Contrastive Learning for Urban Land Cover Classification With Multimodal Siamese Network

Land Use Classification Via Multi-Modal Complementary Feature Fusion and Context Information Enhancement for Optical and Sar Images

Learning Locality-Constrained Sparse Coding for Spectral Enhancement of Multispectral Imagery

Hyperspectral Imagery Classification with Cascaded Support Vector Machines and Multi-Scale Superpixel Segmentation

Superpixel-feature-based multiple kernel sparse representation for hyperspectral image classification

Iterative Semi-Supervised Learning With Few-Shot Samples for Coastal Wetland Land Cover Classification

Self-supervised SAR-optical Data Fusion and Land-cover Mapping using Sentinel-1/-2 Images

Spectral-Spatial Hyperspectral Image Classification With Superpixel Pattern And Extreme Learning Machine