Abstract:Hyperspectral images (HSIs) contain spatially structured information and pixel−level sequential spectral attributes. The continuous spectral features contain hundreds of wavelength bands and the differences between spectra are essential for achieving fine−grained classification. Due to the limited receptive field of backbone networks, convolutional neural networks (CNNs)−based HSI classification methods show limitations in modeling spectral−wise long−range dependencies with fixed kernel size and a limited number of layers. Recently, the self−attention mechanism of transformer framework is introduced to compensate for the limitations of CNNs and to mine the long−term dependencies of spectral signatures. Therefore, many joint CNN and Transformer architectures for HSI classification have been proposed to obtain the merits of both networks. However, these architectures make it difficult to capture spatial–spectral correlation and CNNs distort the continuous nature of the spectral signature because of the over−focus on spatial information, which means that the transformer can easily encounter bottlenecks in modeling spectral−wise similarity and long−range dependencies. To address this problem, we propose a neighborhood enhancement hybrid transformer (NEHT) network. In particular, a simple 2D convolution module is adopted to achieve dimensionality reduction while minimizing the distortion of the original spectral distribution by stacked CNNs. Then, we extract group−wise spatial–spectral features in a parallel design to enhance the representation capability of each token. Furthermore, a feature fusion strategy is introduced to increase subtle discrepancies of spectra. Finally, the self−attention of transformer is employed to mine the long−term dependencies between the enhanced feature sequences. Extensive experiments are performed on three well−known datasets and the proposed NEHT network shows superiority over state−of−the−art (SOTA) methods. Specifically, our proposed method outperforms the SOTA method by 0.46%, 1.05% and 0.75% on average in overall accuracy, average accuracy and kappa coefficient metrics.

Siamese transformer with hierarchical concept embedding for fine-grained image recognition

A Novel Transformer Network with a CNN-Enhanced Cross-Attention Mechanism for Hyperspectral Image Classification

Hierarchical Gate Network for Fine-Grained Visual Recognition.

Siamese Transformer Networks for Few-shot Image Classification

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

Transformer with peak suppression and knowledge guidance for fine-grained image recognition

An Integrated Transformer with Collaborative Tokens Mining for Fine-Grained Recognition

Hybrid Granularities Transformer for Fine-Grained Image Recognition

Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classification

Part-Guided Relational Transformers for Fine-Grained Visual Recognition

Vision Transformers with Hierarchical Attention

Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding

HT-Net: hierarchical context-attention transformer network for medical ct image segmentation

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

Efficient Hybrid Transformer: Learning Global-local Context for Urban Sence Segmentation

Integration Transformer for Ground-Based Cloud Image Segmentation

SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization.

A lightweight siamese transformer for few-shot semantic segmentation

Deep hierarchical transformer for change detection in high-resolution remote sensing images

From Plane to Hierarchy: Deformable Transformer for Remote Sensing Image Captioning

Hyperspectral Image Classification via Spectral Pooling and Hybrid Transformer