Abstract:Despite the success of Transformers in self- supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, but the settings and metrics are different across studies. Trade-off at various compression rates are also largely missing in prior work, making it difficult to compare compression techniques. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report trade- off at various compression rate, including wall-clock time, the number of parameters, and the number of multiply-accumulate operations. Our results show that compared to recent approaches, basic compression techniques are strong baselines. We further present several applications of our results, revealing properties of Transformers, such as the significance of diagonal attention heads. In addition, our results lead to a simple combination of compression techniques that improves trade-off over recent approaches. We hope the results would promote more diverse comparisons among model compression techniques and promote the use of model compression as a tool for analyzing models. Our code of compressing speech self-supervised model is available at <a class="link-external link-https" href="https://github.com/nervjack2/Speech-SSL-Compression/" rel="external noopener nofollow">this https URL</a>.

Part-of-Speech Models Compression Methods for on-Device Grapheme-to-Phoneme Conversion

A COMPRESSION METHOD USED IN LANGUAGE MODELING FOR HANDHELD DEVICES

Language Modeling Is Compression

Evaluating Large Language Models for Generalization and Robustness via Data Compression

A Model Compression Method with Matrix Product Operators for Speech Enhancement

Extreme Model Compression for On-device Natural Language Understanding

Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

Compression of end-to-end non-autoregressive image-to-speech system for low-resourced devices

A Method to Build a Super Small but Practically Accurate Language Model for Handheld Devices

Aggressive Post-Training Compression on Extremely Large Language Models

Exploring compressibility of transformer based text-to-music (TTM) models

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data

A Comprehensive Survey of Compression Algorithms for Language Models

Language-Codec: Reducing the Gaps Between Discrete Codec Representation and Speech Language Models

Compressing Transformer-based self-supervised models for speech processing

Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models

Model Compression for DNN-based Speaker Verification Using Weight Quantization

Intriguing Properties of Compression on Multilingual Models

A Survey on Model Compression for Large Language Models

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

Joint Goal for Word Embedding Compression Based on Word Frequency