Abstract:This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a promising alternative to convolutional neural networks for visual recognition. But when there is no sufficient data, it gets stuck in overfitting and shows inferior performance. To improve data efficiency, we propose hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling and optimize the learnable parameters through latent attribute surrogates. The intrinsic image structure is utilized to reduce the ambiguity between foreground content and background noise by spectral tokens pooling. And the attribute surrogate learning scheme is designed to benefit from the rich visual information in image-label pairs instead of simple visual concepts assigned by their labels. Our Hierarchically Cascaded Transformers, called HCTransformers, is built upon a self-supervised learning framework DINO and is tested on several popular few-shot learning benchmarks. In the inductive setting, HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1-shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet, which demonstrates HCTransformers are efficient to extract discriminative features. Also, HCTransformers show clear advantages over SOTA few-shot classification methods in both 5-way 1-shot and 5-way 5-shot settings on four popular benchmark datasets, including miniImageNet, tieredImageNet, FC100, and CIFAR-FS. The trained weights and codes are available at <a class="link-external link-https" href="https://github.com/StomachCold/HCTransformers" rel="external noopener nofollow">this https URL</a>.

Higher Order Transformers: Efficient Attention Mechanism for Tensor Structured Data

Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data

Tensor Attention Training: Provably Efficient Learning of Higher-order Transformers

HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformers

Axial Attention in Multidimensional Transformers

DARKER: Efficient Transformer with Data-Driven Attention Mechanism for Time Series

Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token Orthogonalization

Adaptive Multi-Resolution Attention with Linear Complexity

Representational Strengths and Limitations of Transformers

Compute-Efficient Medical Image Classification with Softmax-Free Transformers and Sequence Normalization

FAST: Factorizable Attention for Speeding up Transformers

Improving Transformers with Dynamically Composable Multi-Head Attention

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

Generalized Probabilistic Attention Mechanism in Transformers

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Efficient High-Resolution Time Series Classification via Attention Kronecker Decomposition

Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond

Jump Self-attention: Capturing High-order Statistics in Transformers

Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers

The Power of Hard Attention Transformers on Data Sequences: A Formal Language Theoretic Perspective

Transformer Acceleration with Dynamic Sparse Attention