Abstract:Hyperspectral image (HSI) provides rich spectral–spatial information and the light detection and ranging (LiDAR) data reflect the elevation information, which can be jointly exploited for better land-cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image difference, current pixel-wise feature fusion classification methods relying on concatenation or weighted fusion are not effective. To achieve accurate classification result, it is important to extract and fuse similar high-order semantic information and complementary discriminative information contained in multimodal data. In this paper, we propose a novel coupled adversarial learning based classification (CALC) method for fusion classification of HSI and LiDAR data. In specific, a coupled adversarial feature learning (CAFL) sub-network is first trained, to effectively learn the high-order semantic features from HSI and LiDAR data in an unsupervised manner. On one hand, the proposed CAFL sub-network establishes an adversarial game between dual generators and discriminators, so that the learnt features can preserve detail information in HSI and LiDAR data, respectively. On the other hand, by designing weight-sharing and linear fusion structure in the dual generators, we can simultaneously extract similar high-order semantic information and modal-specific complementary information. Meanwhile, a supervised multi-level feature fusion classification (MFFC) sub-network is trained, to further improve the classification performance via adaptive probability fusion strategy. In brief, the low-level, mid-level and high-level features learnt by the CAFL sub-network lead to multiple class estimation probabilities, which are then adaptively combined to generate a final accurate classification result. Both the CAFL and MFFC sub-networks are collaboratively trained by optimizing a designed joint loss function, which consists of unsupervised adversarial loss and supervised classification loss. Overall, by optimizing the joint loss function, the proposed CALC network is pushed to learn highly discriminative fusion features from multimodal data, leading to higher classification accuracies. Extensive experiments on three well-known HSI and LiDAR data sets demonstrate the superior classification performance by the proposed CALC method than several state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/CALC.

Discriminating multiple kernel learning for joint classification of optical and LiDAR data in urban area

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Composite Kernel Local Angular Discriminant Analysis for Multi-Sensor Geospatial Image Analysis

Joint Classification of Hyperspectral and LiDAR Data via Multiprobability Decision Fusion Method

Multiple Kernel Subspace Learning for Clustering and Classification

A novel graph-based multiple kernel learning framework for hyperspectral image classification

Multiple kernel learning via orthogonal neighborhood preserving projection and maximum margin criterion method for synthetic aperture radar target recognition

Discriminative Multiple Canonical Correlation Analysis for Information Fusion

Multi-dimensional deep dense residual networks and multiple kernel learning for hyperspectral image classification

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

Multiple Feature-Based Superpixel-Level Decision Fusion for Hyperspectral and LiDAR Data Classification

DWS-MKL: Depth-width-scaling multiple kernel learning for data classification

A Combination of Lie Group Machine Learning and Deep Learning for Remote Sensing Scene Classification Using Multi-Layer Heterogeneous Feature Extraction and Fusion

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Combined Multi-Kernel Support Vector Machine and Wavelet Analysis for Hyperspectral Remote Sensing Image Classification

Learning in Multimodal and Mixmodal Data: Locality Preserving Discriminant Analysis with Kernel and Sparse Representation Techniques.

A spatial–spectral classification framework for multispectral LiDAR

A Multi-Kernel Mode Using a Local Binary Pattern and Random Patch Convolution for Hyperspectral Image Classification

Assisted learning for land use classification: The important role of semantic correlation between heterogeneous images

Two-stage multiple kernel learning with multiclass kernel polarization.

Group-based local adaptive deep multiple kernel learning with lp norm