Abstract:Remote sensing image classification (RSIC) is a classical and fundamental task in the intelligent interpretation of remote sensing imagery, which can provide unique labeling information for each acquired remote sensing image. Thanks to the potent global context information extraction ability of the multi-head self-attention (MSA) mechanism, visual transformer (ViT)-based architectures have shown excellent capability in natural scene image classification. However, in order to achieve powerful RSIC performance, it is insufficient to capture global spatial information alone. Specifically, for fine-grained target recognition tasks with high inter-class similarity, discriminative and effective local feature representations are key to correct classification. In addition, due to the lack of inductive biases, the powerful global spatial context representation capability of ViT requires lengthy training procedures and large-scale pre-training data volume. To solve the above problems, a hybrid architecture of convolution neural network (CNN) and ViT is proposed to improve the RSIC ability, called P2FEViT, which integrates plug-and-play CNN features with ViT. In this paper, the feature representation capabilities of CNN and ViT applying for RSIC are first analyzed. Second, aiming to integrate the advantages of CNN and ViT, a novel approach embedding CNN features into the ViT architecture is proposed, which can make the model synchronously capture and fuse global context and local multimodal information to further improve the classification capability of ViT. Third, based on the hybrid structure, only a simple cross-entropy loss is employed for model training. The model can also have rapid and comfortable convergence with relatively less training data than the original ViT. Finally, extensive experiments are conducted on the public and challenging remote sensing scene classification dataset of NWPU-RESISC45 (NWPU-R45) and the self-built fine-grained target classification dataset called BIT-AFGR50. The experimental results demonstrate that the proposed P2FEViT can effectively improve the feature description capability and obtain outstanding image classification performance, while significantly reducing the high dependence of ViT on large-scale pre-training data volume and accelerating the convergence speed. The code and self-built dataset will be released at our webpages.

A shape-aware enhancement Vision Transformer for building extraction from remote sensing imagery

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

Extracting Building Footprint From Remote Sensing Images by an Enhanced Vision Transformer Network

Building Extraction With Vision Transformer

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

SDSC-UNet: Dual Skip Connection ViT-Based U-Shaped Model for Building Extraction

MSFTrans: a multi-task frequency-spatial learning transformer for building extraction from high spatial resolution remote sensing images

DSAT-Net: Dual Spatial Attention Transformer for Building Extraction From Aerial Images

SCViT: A Spatial-Channel Feature Preserving Vision Transformer for Remote Sensing Image Scene Classification

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model

MPViT: Multi-Path Vision Transformer for Dense Prediction

P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification

Asymmetric Network Combining CNN and Transformer for Building Extraction from Remote Sensing Images

Vectorized Building Extraction from High-Resolution Remote Sensing Images Using Spatial Cognitive Graph Convolution Model

A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation

Transformer-Based Semantic Segmentation for Extraction of Building Footprints from Very-High-Resolution Images

DAT++: Spatially Dynamic Vision Transformer with Deformable Attention

SegViT: Semantic Segmentation with Plain Vision Transformers