Abstract:With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM

Towards a Deeper Understanding of Transformer for Residential Non-intrusive Load Monitoring

Nonintrusive Load Monitoring (NILM) Using a Deep Learning Model with a Transformer-Based Attention Mechanism and Temporal Pooling

Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring

Sequence-to-Sequence Model with Transformer-based Attention Mechanism and Temporal Pooling for Non-Intrusive Load Monitoring

Escaping the Big Data Paradigm with Compact Transformers

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

DCML: Boosting Applying Experience of NILM with Dilated Convolution and Multi-Task Learning

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition

Enhancing Automatic Modulation Recognition for IoT Applications Using Transformers

Disentangling and Integrating Relational and Sensory Information in Transformer Architectures

Transformer-Aided Semantic Communications

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices

Homeostasis and Sparsity in Transformer

Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation

Reducing the Transformer Architecture to a Minimum

Transformer-BLS: An efficient learning algorithm based on multi-head attention mechanism and incremental learning algorithms

Trainable Transformer in Transformer

Few-Shot Class Incremental Learning via Robust Transformer Approach