Abstract:Remote sensing image classification (RSIC) is a classical and fundamental task in the intelligent interpretation of remote sensing imagery, which can provide unique labeling information for each acquired remote sensing image. Thanks to the potent global context information extraction ability of the multi-head self-attention (MSA) mechanism, visual transformer (ViT)-based architectures have shown excellent capability in natural scene image classification. However, in order to achieve powerful RSIC performance, it is insufficient to capture global spatial information alone. Specifically, for fine-grained target recognition tasks with high inter-class similarity, discriminative and effective local feature representations are key to correct classification. In addition, due to the lack of inductive biases, the powerful global spatial context representation capability of ViT requires lengthy training procedures and large-scale pre-training data volume. To solve the above problems, a hybrid architecture of convolution neural network (CNN) and ViT is proposed to improve the RSIC ability, called P2FEViT, which integrates plug-and-play CNN features with ViT. In this paper, the feature representation capabilities of CNN and ViT applying for RSIC are first analyzed. Second, aiming to integrate the advantages of CNN and ViT, a novel approach embedding CNN features into the ViT architecture is proposed, which can make the model synchronously capture and fuse global context and local multimodal information to further improve the classification capability of ViT. Third, based on the hybrid structure, only a simple cross-entropy loss is employed for model training. The model can also have rapid and comfortable convergence with relatively less training data than the original ViT. Finally, extensive experiments are conducted on the public and challenging remote sensing scene classification dataset of NWPU-RESISC45 (NWPU-R45) and the self-built fine-grained target classification dataset called BIT-AFGR50. The experimental results demonstrate that the proposed P2FEViT can effectively improve the feature description capability and obtain outstanding image classification performance, while significantly reducing the high dependence of ViT on large-scale pre-training data volume and accelerating the convergence speed. The code and self-built dataset will be released at our webpages.

Plant and Animal Species Recognition Based on Dynamic Vision Transformer Architecture

DSViT: Dynamically Scalable Vision Transformer for Remote Sensing Image Segmentation and Classification.

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

Deep Hierarchical Vision Transformer for Hyperspectral and LiDAR Data Classification

A free lunch from ViT:Adaptive Attention Multi-scale Fusion Transformer for Fine-grained Visual Recognition

ZoomViT: an Observation Behavior-Based Fine-Grained Recognition Scheme

MViT: Mask Vision Transformer for Facial Expression Recognition in the Wild

Extracting Building Footprint From Remote Sensing Images by an Enhanced Vision Transformer Network

Ts-vit: feature-enhanced transformer via token selection for fine-grained image recognition

DAT++: Spatially Dynamic Vision Transformer with Deformable Attention

Human Body Segmentation in Wide-Angle Images Based on Fast Vision Transformers

DenseViT: A Hybrid CNN-Vision Transformer Model for an Improved Multisensor Lithological Classification

A FREE LUNCH FROM VIT: ADAPTIVE ATTENTION MULTI-SCALE FUSION TRANSFORMER FOR FINE-GRAINED VISUAL RECOGNITION

A Novel Hybrid Transformer-Cnn Architecture for Environmental Microorganism Classification.

P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification

Vision Transformers: From Semantic Segmentation to Dense Prediction

ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connections

Discriminative Vision Transformer for Heterogeneous Cross-Domain Hyperspectral Image Classification

DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition

Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series

Data Augmentation Vision Transformer for Fine-grained Image Classification