Abstract:The recent success of attention mechanism-driven deep models, like vision transformer (ViT) as one of the most representatives, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, aiming at the task of land use and land cover (LULC) classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention (CMA) module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full token-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral (HS) and light detection and ranging (LiDAR) data, and Berlin dataset with HS and synthetic aperture radar (SAR) data, to demonstrate that our extended vision transformer (ExViT) outperforms concurrent competitors based on transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine-learning-based models. The source codes and investigated datasets of this work will be made publicly a- ailable at https://github.com/jingyao16/ExViT.

Leveraging Large Vision Models for Terrain Classification in HiRISE Mars Orbital Images

Recent advances in the application of vision transformers to remote sensing image scene classification

Vision Transformer for Multispectral Satellite Imagery: Advancing Landcover Classification*

Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model

Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Models

Implementing vision transformer for classifying 2D biomedical images

Automated classification of remote sensing satellite images using deep learning based vision transformer

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

StrideNET: Swin Transformer for Terrain Recognition with Dynamic Roughness Extraction

Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images

Image-Based Lunar Hazard Detection in Low Illumination Simulated Conditions via Vision Transformers

Multivariate image processing in minerals engineering with vision transformers

Deep Hierarchical Vision Transformer for Hyperspectral and LiDAR Data Classification

Vision Transformers, a new approach for high-resolution and large-scale mapping of canopy heights

On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery

Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model

ViTAR: Vision Transformer with Any Resolution

HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs

Vision Transformers for Remote Sensing Image Classification