Abstract:The recent success of attention mechanism-driven deep models, like vision transformer (ViT) as one of the most representatives, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, aiming at the task of land use and land cover (LULC) classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention (CMA) module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full token-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral (HS) and light detection and ranging (LiDAR) data, and Berlin dataset with HS and synthetic aperture radar (SAR) data, to demonstrate that our extended vision transformer (ExViT) outperforms concurrent competitors based on transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine-learning-based models. The source codes and investigated datasets of this work will be made publicly a- ailable at https://github.com/jingyao16/ExViT.

Transformer-Based Incomplete Multi-Modal Learning for Land Cover Classification

Multiscale 3-D-2-D Mixed CNN and Lightweight Attention-Free Transformer for Hyperspectral and LiDAR Classification

Incomplete Multimodal Learning for Remote Sensing Data Fusion

Transformer-based Contrastive Prototypical Clustering for Multimodal Remote Sensing Data.

Multimodal Fusion Transformer for Remote Sensing Image Classification

A Novel Approach to Incomplete Multimodal Learning for Remote Sensing Data Fusion

Introducing Improved Transformer to Land Cover Classification Using Multispectral LiDAR Point Clouds

Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

Local Climate Zone Classification via Semi-Supervised Multimodal Multiscale Transformer

TCPSNet: Transformer and Cross-Pseudo-Siamese Learning Network for Classification of Multi-Source Remote Sensing Images

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Learning transformer-based heterogeneously salient graph representation for multimodal remote sensing image classification

Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

Joint Classification of Hyperspectral and LiDAR Data Based on Adaptive Gating Mechanism and Learnable Transformer

MCFT: Multimodal Contrastive Fusion Transformer for Classification of Hyperspectral Image and LiDAR Data

Attention Fusion of Transformer-Based and Scale-Based Method for Hyperspectral and LiDAR Joint Classification

Meta-Transformer: A Unified Framework for Multimodal Learning

Transformer-based land use and land cover classification with explainability using satellite imagery

A Lightweight Transformer With Multigranularity Tokens and Connected Component Loss for Land Cover Classification

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

Robust Land Cover Classification with Multimodal Knowledge Distillation