Abstract:In this paper, we introduce the <a class="link-external link-http" href="http://big.LITTLE" rel="external noopener nofollow">this http URL</a> Vision Transformer, an innovative architecture aimed at achieving efficient visual recognition. This dual-transformer system is composed of two distinct blocks: the big performance block, characterized by its high capacity and substantial computational demands, and the LITTLE efficiency block, designed for speed with lower capacity. The key innovation of our approach lies in its dynamic inference mechanism. When processing an image, our system determines the importance of each token and allocates them accordingly: essential tokens are processed by the high-performance big model, while less critical tokens are handled by the more efficient little model. This selective processing significantly reduces computational load without sacrificing the overall performance of the model, as it ensures that detailed analysis is reserved for the most important information. To validate the effectiveness of our <a class="link-external link-http" href="http://big.LITTLE" rel="external noopener nofollow">this http URL</a> Vision Transformer, we conducted comprehensive experiments on image classification and segment anything task. Our results demonstrate that the <a class="link-external link-http" href="http://big.LITTLE" rel="external noopener nofollow">this http URL</a> architecture not only maintains high accuracy but also achieves substantial computational savings. Specifically, our approach enables the efficient handling of large-scale visual recognition tasks by dynamically balancing the trade-offs between performance and efficiency. The success of our method underscores the potential of hybrid models in optimizing both computation and performance in visual recognition tasks, paving the way for more practical and scalable deployment of advanced neural networks in real-world applications.

Enhanced Vision Transformer with Dual-Dimensional Self-Attention for Image Recognition

FAM: Improving columnar vision transformer with feature attention mechanism

Lite Vision Transformer with Enhanced Self-Attention

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

big.LITTLE Vision Transformer for Efficient Visual Recognition

Glance-and-Gaze Vision Transformer

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

TripleFormer: improving transformer-based image classification method using multiple self-attention inputs

Improving Vision Transformers by Overlapping Heads in Multi-Head Self-Attention

Vision Transformer with Convolutions Architecture Search

ResT: an Efficient Transformer for Visual Recognition

Armour: Generalizable Compact Self-Attention for Vision Transformers

Data Augmentation Vision Transformer for Fine-grained Image Classification

Dual Path Transformer with Partition Attention

Vicinity Vision Transformer

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

ACC-ViT : Atrous Convolution's Comeback in Vision Transformers

Dual-Dependency Attention Transformer for Fine-Grained Visual Classification