Abstract:The recent success of attention mechanism-driven deep models, like vision transformer (ViT) as one of the most representatives, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, aiming at the task of land use and land cover (LULC) classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention (CMA) module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full token-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral (HS) and light detection and ranging (LiDAR) data, and Berlin dataset with HS and synthetic aperture radar (SAR) data, to demonstrate that our extended vision transformer (ExViT) outperforms concurrent competitors based on transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine-learning-based models. The source codes and investigated datasets of this work will be made publicly a- ailable at https://github.com/jingyao16/ExViT.

Reperceive Global Vision of Transformer for Remote Sensing Images Weakly Supervised Object Localization

LCTR: On Awakening the Local Continuity of Transformer for Weakly Supervised Object Localization

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

HiCT: Hierarchical Comprehend of Transformer for Weakly Supervised Object Localization

SemiCVT: Semi-Supervised Convolutional Vision Transformer for Semantic Segmentation

Weakly Supervised Object Localization Using Long-Range Semantic Foreground Activation

Semantic-Constraint Matching Transformer for Weakly Supervised Object Localization

Task-Aware Weakly Supervised Object Localization With Transformer

Enhanced Window-Based Self-Attention with Global and Multi-Scale Representations for Remote Sensing Image Super-Resolution

Spatial-Aware Token for Weakly Supervised Object Localization

Unifying Global-Local Representations in Salient Object Detection with Transformer

Re-Attention Transformer for Weakly Supervised Object Localization

Proxy Probing Decoder for Weakly Supervised Object Localization: A Baseline Investigation

Leveraging Transformers for Weakly Supervised Object Localization in Unconstrained Videos

Discriminative Sampling of Proposals in Self-Supervised Transformers for Weakly Supervised Object Localization

Salient Object Detection in Optical Remote Sensing Images Driven by Transformer

Locality-Enhanced Transformer for Semantic Segmentation of High-Resolution Remote Sensing Images.

SCViT: A Spatial-Channel Feature Preserving Vision Transformer for Remote Sensing Image Scene Classification

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

EGSA: Enhanced and Global Semantic Activation for Weakly Supervised Object Localization.

RegionViT: Regional-to-Local Attention for Vision Transformers