Abstract:The recent success of attention mechanism-driven deep models, like vision transformer (ViT) as one of the most representatives, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, aiming at the task of land use and land cover (LULC) classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention (CMA) module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full token-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral (HS) and light detection and ranging (LiDAR) data, and Berlin dataset with HS and synthetic aperture radar (SAR) data, to demonstrate that our extended vision transformer (ExViT) outperforms concurrent competitors based on transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine-learning-based models. The source codes and investigated datasets of this work will be made publicly a- ailable at https://github.com/jingyao16/ExViT.

SCENIC: A JAX Library for Computer Vision Research and Beyond

MegaScenes: Scene-Level View Synthesis at Scale

Scenic: a language for scenario specification and data generation

Vision Transformer with Sparse Scan Prior

GzScenic: Automatic Scene Generation for Gazebo Simulator

VistaFormer: Scalable Vision Transformers for Satellite Image Time Series Segmentation

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model

Scattering Vision Transformer: Spectral Mixing Matters

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image

SkyScenes: A Synthetic Dataset for Aerial Scene Understanding

Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers

Edge-Assisted Epipolar Transformer for Industrial Scene Reconstruction

SynJax: Structured Probability Distributions for JAX

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

Bioinspired Scene Classification by Deep Active Learning With Remote Sensing Applications

SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

SRCN3D: Sparse R-CNN 3D for Compact Convolutional Multi-View 3D Object Detection and Tracking

HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

ProtSCAPE: Mapping the landscape of protein conformations in molecular dynamics