Abstract:This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a promising alternative to convolutional neural networks for visual recognition. But when there is no sufficient data, it gets stuck in overfitting and shows inferior performance. To improve data efficiency, we propose hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling and optimize the learnable parameters through latent attribute surrogates. The intrinsic image structure is utilized to reduce the ambiguity between foreground content and background noise by spectral tokens pooling. And the attribute surrogate learning scheme is designed to benefit from the rich visual information in image-label pairs instead of simple visual concepts assigned by their labels. Our Hierarchically Cascaded Transformers, called HCTransformers, is built upon a self-supervised learning framework DINO and is tested on several popular few-shot learning benchmarks. In the inductive setting, HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1-shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet, which demonstrates HCTransformers are efficient to extract discriminative features. Also, HCTransformers show clear advantages over SOTA few-shot classification methods in both 5-way 1-shot and 5-way 5-shot settings on four popular benchmark datasets, including miniImageNet, tieredImageNet, FC100, and CIFAR-FS. The trained weights and codes are available at <a class="link-external link-https" href="https://github.com/StomachCold/HCTransformers" rel="external noopener nofollow">this https URL</a>.

Csat: Contrastive Sampling-Aggregating Transformer for Attribute-Missing Graph Learning

Active Sampling for Node Attribute Completion on Graphs

Full-Attention Driven Graph Contrastive Learning: with Effective Mutual Information Insight

AmGCL: Feature Imputation of Attribute Missing Graph via Self-supervised Contrastive Learning

Hierarchical Graph Transformer with Adaptive Node Sampling

Unsupervised Graph Transformer with Augmentation-Free Contrastive Learning

Inductive Attributed Community Search: To Learn Communities Across Graphs

Learning the Implicit Semantic Representation on Graph-Structured Data

EGCT: Enhanced Graph Convolutional Transformer for 3D Point Cloud Representation Learning

Attribute graph clustering via transformer and graph attention autoencoder

SAGES: Scalable Attributed Graph Embedding with Sampling for Unsupervised Learning

AGHINT: Attribute-Guided Representation Learning on Heterogeneous Information Networks with Transformer

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

Transforming Graphs for Enhanced Attribute Clustering: An Innovative Graph Transformer-Based Method

Boosting Graph Contrastive Learning Via Adaptive Sampling

Attribute-Missing Graph Clustering Network

ANGraph: Attribute-Interactive Neighborhood-Aggregative Graph Representation Learning

HeGAE-AC: Heterogeneous graph auto-encoder for attribute completion

GTC: GNN-Transformer Co-contrastive Learning for Self-supervised Heterogeneous Graph Representation

Self-supervised Contrastive Attributed Graph Clustering

Scalable Deep Metric Learning on Attributed Graphs