Abstract:Recently, several Vision Transformer (ViT) based methods have been proposed for Fine-Grained Visual Classification (FGVC). These methods significantly surpass existing CNN-based ones, demonstrating the effectiveness of ViT in FGVC tasks. However, there are some limitations when applying ViT directly to FGVC. First, ViT needs to split images into patches and calculate the attention of every pair, which may result in heavy noise calculation during the training phase and unsatisfying performance when handling fine-grained images with complex backgrounds and small objects. Second, complementary information is important for FGVC, but a standard ViT works by using the class token in the final layer for classification which is not enough to extract comprehensive fine-grained information at different levels. Third, the class token fuses the information of all patches in the same manner, in other words, the class token treats each patch equally. However, the discriminative parts should be more critical. To address these issues, we propose ACC-ViT including three novel components, i.e., Attention Patch Combination (APC), Critical Regions Filter (CRF), and Complementary Tokens Integration (CTI). Thereinto, APC pieces informative patches from two images to generate a new image to mitigate the noisy calculation and reinforce the differences between images. CRF emphasizes tokens corresponding to discriminative regions to generate a new class token for subtle feature learning. To extract comprehensive information, CTI integrates complementary information captured by class tokens in different ViT layers. We conduct comprehensive experiments on four widely used datasets and the results demonstrate that ACC-ViT can achieve competitive performance. The source code is available at https://github.com/Hector0426/fine-grained-image-classification-with-vit.

Efficient Vision Transformer with Token-Selective and Merging Strategies for Autonomous Underwater Vehicles

Fish Detection and Classification Based on Improved ViT

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Underwater image enhancement using lightweight vision transformer

LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation

Efficient Vision Transformer for Human-Centric AIoT Applications Through Token Tracking Assignment

Ts-vit: feature-enhanced transformer via token selection for fine-grained image recognition

So-ViT: Mind Visual Tokens for Vision Transformer

On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery

Making Vision Transformers Efficient from A Token Sparsification View

Efficient Vision Transformer via Token Merger

Multi-Scale And Token Mergence: Make Your ViT More Efficient

Super Vision Transformer

SCViT: A Spatial-Channel Feature Preserving Vision Transformer for Remote Sensing Image Scene Classification

ViTAR: Vision Transformer with Any Resolution

Vision Transformer with Super Token Sampling

DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusion

Scalable Vision Transformers with Hierarchical Pooling.

TPC-ViT: Token Propagation Controller for Efficient Vision Transformer

A Vision Transformer for Fine-Grained Classification by Reducing Noise and Enhancing Discriminative Information

Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers