Abstract:Transformer holds significance in deep learning (DL) research. Node embedding (NE) and positional encoding (PE) are usually two indispensable components in a Transformer. The former can excavate hidden correlations from the data, while the latter can store locational relationships between nodes. Recently, the Transformer has been applied for hyperspectral image (HSI) classification because the model can capture long-range dependencies to aggregate global features for representation learning. In an HSI, adjacent pixels tend to be homogeneous, while the NE does not identify the positional information of pixels. Therefore, PE is crucial for Transformers to understand locational relationships between pixels. However, in this area, most Transformer-based methods randomly generate PEs without considering their physical meaning, which leads to weak representations. This article proposes a new graph generative structure-aware Transformer (GraphGST) to solve the above-mentioned PE problem when implementing HSI classification. In our GraphGST, a new absolute PE (APE) is established to acquire pixels' absolute positional sequences (APSs) and is integrated into the Transformer architecture. Moreover, a generative mechanism with self-supervised learning is developed to achieve cross-view contrastive learning (CL), aiming to enhance the representation learning of the Transformer. The proposed GraphGST model can capture local-to-global correlations, and the extracted APSs can complement the spectral features of pixels to assist in NE. Several experiments with real HSIs are conducted to evaluate the effectiveness of our GraphGST. The proposed method demonstrates very competitive performance compared with other state-of-the-art (SOTA) approaches. Our source codes will be provided in the following link https://github.com/yuanchaosu/TGRS-graphGST.

Technical Report: The Graph Spectral Token -- Enhancing Graph Transformers with Spectral Information

GTA: Graph Transformer Adapter

Learning Graph Quantized Tokenizers for Transformers

A graph-guided transformer based on dual-stream perception for hyperspectral image classification

Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers

Unleashing the Power of Transformer for Graphs

Graph Transformers: A Survey

Reach the Remote Neighbors: Dual-Encoding Transformer for Graphs

On Structural Expressive Power of Graph Transformers

SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations

Spectral Transform Forms Scalable Transformer

NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification

SignGT: Signed Attention-based Graph Transformer for Graph Representation Learning

Attending to Graph Transformers

Less is More: on the Over-Globalizing Problem in Graph Transformers

GraphGST: Graph Generative Structure-Aware Transformer for Hyperspectral Image Classification

Hybrid Focal and Full-Range Attention Based Graph Transformers

Transformer for Graphs: An Overview from Architecture Perspective

A Pure Transformer Pretraining Framework on Text-attributed Graphs

TorchGT: A Holistic System for Large-scale Graph Transformer Training

DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability