Abstract:Remote sensing image classification (RSIC) is a classical and fundamental task in the intelligent interpretation of remote sensing imagery, which can provide unique labeling information for each acquired remote sensing image. Thanks to the potent global context information extraction ability of the multi-head self-attention (MSA) mechanism, visual transformer (ViT)-based architectures have shown excellent capability in natural scene image classification. However, in order to achieve powerful RSIC performance, it is insufficient to capture global spatial information alone. Specifically, for fine-grained target recognition tasks with high inter-class similarity, discriminative and effective local feature representations are key to correct classification. In addition, due to the lack of inductive biases, the powerful global spatial context representation capability of ViT requires lengthy training procedures and large-scale pre-training data volume. To solve the above problems, a hybrid architecture of convolution neural network (CNN) and ViT is proposed to improve the RSIC ability, called P2FEViT, which integrates plug-and-play CNN features with ViT. In this paper, the feature representation capabilities of CNN and ViT applying for RSIC are first analyzed. Second, aiming to integrate the advantages of CNN and ViT, a novel approach embedding CNN features into the ViT architecture is proposed, which can make the model synchronously capture and fuse global context and local multimodal information to further improve the classification capability of ViT. Third, based on the hybrid structure, only a simple cross-entropy loss is employed for model training. The model can also have rapid and comfortable convergence with relatively less training data than the original ViT. Finally, extensive experiments are conducted on the public and challenging remote sensing scene classification dataset of NWPU-RESISC45 (NWPU-R45) and the self-built fine-grained target classification dataset called BIT-AFGR50. The experimental results demonstrate that the proposed P2FEViT can effectively improve the feature description capability and obtain outstanding image classification performance, while significantly reducing the high dependence of ViT on large-scale pre-training data volume and accelerating the convergence speed. The code and self-built dataset will be released at our webpages.

Improved Prototypical Networks for Remote Sensing Scene Classification

Remote Sensing Scene Classification by Gated Bidirectional Network

SPNet: Siamese-Prototype Network for Few-Shot Remote Sensing Image Scene Classification

P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification

Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification

ALPN: Active-Learning-Based Prototypical Network for Few-Shot Hyperspectral Imagery Classification

Remote Sensing Object Detection in Disturbed Environment Based on Improved RTMDet Network

Object Detection in Remote Sensing Imagery Based on Prototype Learning Network With Proposal Relation

Heterogeneous Prototype Distillation With Support-Query Correlative Guidance for Few-Shot Remote Sensing Scene Classification

RS-SSKD: Self-Supervision Equipped with Knowledge Distillation for Few-Shot Remote Sensing Scene Classification

PSCLI-TF: Position-Sensitive Cross-Layer Interactive Transformer Model for Remote Sensing Image Scene Classification

Improved Prototypical Network Model for Forest Species Classification in Complex Stand

RS-MetaNet: Deep meta metric learning for few-shot remote sensing scene classification

InfRS: Incremental Few-Shot Object Detection in Remote Sensing Images

Remote Sensing Image Scene Classification via Graph Template Enhancement and Supplementation Network With Dual-Teacher Knowledge Distillation

Multi-pretext-task Prototypes Guided Dynamic Contrastive Learning Network for Few-shot Remote Sensing Scene Classification

Transferring CNN With Adaptive Learning for Remote Sensing Scene Classification

Learn by Yourself: A Feature-Augmented Self-Distillation Convolutional Neural Network for Remote Sensing Scene Image Classification

L2-norm prototypical networks for tackling the data shift problem in scene classification

Representation-Enhanced Status Replay Network for Multisource Remote-Sensing Image Classification

A Rotation Invariance Spatial Transformation Network for Remote Sensing Image Retrieval