Abstract:Fine-grained recognition mainly classifies subclass images into hundreds of subcategorical labels by locating the discriminative regions (e.g., Cape May warbler or Magnolia warbler bird). Due to the high complexity and non-differentiation of region locations through the traditional backbone architecture, most existing approaches utilize multi-level reinforcement learning to distinguish the similar appearance among sub-categories. These methods explore incomplete information through only the intra-class informative regions in one image or the inter-class and intra-class relationship in pairwise images, leading to the tendency for overlapped region locations. Since the inter-class correlations and new backbone with complete contextual semantic information play important roles in distinguishing fine-grained classes, we propose a novel transformer with the collaborative token mining (TCTM) scheme by fully exploiting the relationships between inter-class and intra-class regions. The proposed TCTM scheme with a new transformer backbone consists of two modules that collaboratively explore the spatially aware tokens: the Pyramid Tokens Multiplication (PTM) module which exploits the integrated multi-stage inter-class and intra-class correlations from new transformer architecture and the Tokens Proposals Generation (TPG) module which captures two groups of top-four discriminative tokens. The two PTMs extract contrastive tokens for each image and learn to rank these tokens, assuming that those from the same class and the same module should have smaller distances. The TPGs further sort and update the candidate tokens from the extracted attention tokens by ranking their probabilities with ground truth subcategorical labels. Through the collaboration between the PTM and TPG, our TCTM scheme can take the integrated correlations into account and mine the discriminative tokens for final fine-grained classification. Extensive experiments on four popular benchmarks show that our proposed TCTM outperforms the state-of-the-art methods by a large margin.

Synergistically Learning Class-specific Tokens for Multi-class Whole Slide Image Classification

Multi-Scale Prototypical Transformer for Whole Slide Image Classification

Transformer-Based Video-Structure Multi-Instance Learning for Whole Slide Image Classification

CWC-transformer: a visual transformer approach for compressed whole slide image classification

Multi-class Cancer Classification of Whole Slide Images Through Transformer and Multiple Instance Learning.

Multi-level Multiple Instance Learning with Transformer for Whole Slide Image Classification

Multi-scale Efficient Graph-Transformer for Whole Slide Image Classification

Unsupervised mutual transformer learning for multi-gigapixel whole slide image classification

MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation

Trusted multi-scale classification framework for whole slide image

Enhancing Whole Slide Image Classification with Discriminative and Contrastive Learning

MG-Trans: Multi-Scale Graph Transformer with Information Bottleneck for Whole Slide Image Classification.

Patch Transformer for Multi-tagging Whole Slide Histopathology Images

Shared-Specific Feature Learning with Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.

Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics

A Multi-Range Spectral-Spatial Transformer for Hyperspectral Image Classification

An Integrated Transformer with Collaborative Tokens Mining for Fine-Grained Recognition

Multilevel Class Token Transformer With Cross TokenMixer for Hyperspectral Images Classification

A self-supervised framework for learning whole slide representations

Multi-Part Token Transformer with Dual Contrastive Learning for Fine-grained Image Classification