Abstract:Recently, several Vision Transformer (ViT) based methods have been proposed for Fine-Grained Visual Classification (FGVC). These methods significantly surpass existing CNN-based ones, demonstrating the effectiveness of ViT in FGVC tasks. However, there are some limitations when applying ViT directly to FGVC. First, ViT needs to split images into patches and calculate the attention of every pair, which may result in heavy noise calculation during the training phase and unsatisfying performance when handling fine-grained images with complex backgrounds and small objects. Second, complementary information is important for FGVC, but a standard ViT works by using the class token in the final layer for classification which is not enough to extract comprehensive fine-grained information at different levels. Third, the class token fuses the information of all patches in the same manner, in other words, the class token treats each patch equally. However, the discriminative parts should be more critical. To address these issues, we propose ACC-ViT including three novel components, i.e., Attention Patch Combination (APC), Critical Regions Filter (CRF), and Complementary Tokens Integration (CTI). Thereinto, APC pieces informative patches from two images to generate a new image to mitigate the noisy calculation and reinforce the differences between images. CRF emphasizes tokens corresponding to discriminative regions to generate a new class token for subtle feature learning. To extract comprehensive information, CTI integrates complementary information captured by class tokens in different ViT layers. We conduct comprehensive experiments on four widely used datasets and the results demonstrate that ACC-ViT can achieve competitive performance. The source code is available at https://github.com/Hector0426/fine-grained-image-classification-with-vit.

RegionViT: Regional-to-Local Attention for Vision Transformers

Fusion of regional and sparse attention in Vision Transformers

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Vision Transformer with Super Token Sampling

LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition

Vicinity Vision Transformer

DctViT: Discrete Cosine Transform Meet Vision Transformers

Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight

Vision Transformer: Vit and its Derivatives

Global Context Vision Transformers

RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition

Rethinking Vision Transformer Through Human–object Interaction Detection

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

SCViT: A Spatial-Channel Feature Preserving Vision Transformer for Remote Sensing Image Scene Classification

BViT: Broad Attention based Vision Transformer

So-ViT: Mind Visual Tokens for Vision Transformer

Vision Transformer with Sparse Scan Prior

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

A Vision Transformer for Fine-Grained Classification by Reducing Noise and Enhancing Discriminative Information

ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connections