Abstract:Transformer has been widely applied in image processing tasks as a substitute for convolutional neural networks (CNNs) for feature extraction due to its superiority in global context modeling and flexibility in model generalization. However, the existing transformer-based methods for semantic segmentation of remote sensing (RS) images are still with several limitations, which can be summarized into two main aspects: 1) the transformer encoder is generally combined with CNN-based decoder, leading to inconsistency in feature representations; and 2) the strategies for global and local context information utilization are not sufficiently effective. Therefore, in this article, a global-local transformer segmentor (GLOTS) framework is proposed for the semantic segmentation of RS images to acquire consistent feature representations by adopting transformers for both encoding and decoding, in which a masked image modeling (MIM) pretrained transformer encoder is adopted to learn semantic-rich representations of input images and a multiscale global-local transformer decoder is designed to fully exploit the global and local features. Specifically, the transformer decoder uses a feature separation-aggregation module (FSAM) to utilize the feature adequately at different scales and adopts a global-local attention module (GLAM) containing global attention block (GAB) and local attention block (LAB) to capture the global and local context information, respectively. Furthermore, a learnable progressive upsampling strategy (LPUS) is proposed to restore the resolution progressively, which can flexibly recover the fine-grained details in the upsampling process. The experiment results on the three benchmark RS datasets demonstrate that the proposed GLOTS is capable of achieving better performance with some state-of-the-art methods, and the superiority of the proposed framework is also verified by ablation studies. The code will be available at https://github.com/lyhnsn/GLOTS.

Sparse and Hierarchical Transformer for Survival Analysis on Whole Slide Images

Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics

Integration of Patch Features Through Self-supervised Learning and Transformer for Survival Analysis on Whole Slide Images.

Generating Hypergraph-Based High-Order Representations of Whole-Slide Histopathological Images for Survival Prediction

Multi-scope Analysis Driven Hierarchical Graph Transformer for Whole Slide Image Based Cancer Survival Prediction

Transformer-Based Multimodal Fusion for Survival Prediction by Integrating Whole Slide Images, Clinical, and Genomic Data

HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image

Transformer-Based Video-Structure Multi-Instance Learning for Whole Slide Image Classification

Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis

HIGT: Hierarchical Interaction Graph-Transformer for Whole Slide Image Analysis

PATHS: A Hierarchical Transformer for Efficient Whole Slide Image Analysis

CWC-transformer: a visual transformer approach for compressed whole slide image classification

Explainable survival analysis with uncertainty using convolution-involved vision transformer

Big-Hypergraph Factorization Neural Network for Survival Prediction From Whole Slide Image

Multi-level Multiple Instance Learning with Transformer for Whole Slide Image Classification

Rethinking Transformers for Semantic Segmentation of Remote Sensing Images.

Multi-Scale Prototypical Transformer for Whole Slide Image Classification

What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning

An Explainable Coarse-to-Fine Survival Analysis Method on Multi-Center Whole Slide Images.

Synergistically Learning Class-specific Tokens for Multi-class Whole Slide Image Classification