Abstract:Although large pre-trained models of code have delivered significant advancements in various code processing tasks, there is an impediment to the wide and fluent adoption of these powerful models in software developers' daily workflow: these large models consume hundreds of megabytes of memory and run slowly on personal devices, which causes problems in model deployment and greatly degrades the user experience. It motivates us to propose Compressor, a novel approach that can compress the pre-trained models of code into extremely small models with negligible performance sacrifice. Our proposed method formulates the design of tiny models as simplifying the pre-trained model architecture: searching for a significantly smaller model that follows an architectural design similar to the original pre-trained model. Compressor proposes a genetic algorithm (GA)-based strategy to guide the simplification process. Prior studies found that a model with higher computational cost tends to be more powerful. Inspired by this insight, the GA algorithm is designed to maximize a model's Giga floating-point operations (GFLOPs), an indicator of the model computational cost, to satisfy the constraint of the target model size. Then, we use the knowledge distillation technique to train the small model: unlabelled data is fed into the large model and the outputs are used as labels to train the small model. We evaluate Compressor with two state-of-the-art pre-trained models, i.e., CodeBERT and GraphCodeBERT, on two important tasks, i.e., vulnerability prediction and clone detection. We use our method to compress pre-trained models to a size (3 MB), which is 160$\times$ smaller than the original size. The results show that compressed CodeBERT and GraphCodeBERT are 4.31$\times$ and 4.15$\times$ faster than the original model at inference, respectively. More importantly, ...

Compressed models are NOT miniature versions of large models

Compressing Pre-trained Models of Code into 3 MB

Improved Model Compression Method Based on Information Entropy

Application Specific Compression of Deep Learning Models

Efficient Model Compression for Bayesian Neural Networks

Pruning at a Glance: Global Neural Pruning for Model Compression

Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference

Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study

Model compression as constrained optimization, with application to neural nets. Part V: combining compressions

Language Modeling Is Compression

What Do Compressed Deep Neural Networks Forget?

Accuracy is Not All You Need

Spear: Evaluate the Adversarial Robustness of Compressed Neural Models

Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment

Are Compressed Language Models Less Subgroup Robust?

ZipNN: Lossless Compression for AI Models

Lossless and Near-Lossless Compression for Foundation Models

Deep learning model compression using network sensitivity and gradients

VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch

Estimation-Based Strategy Generation for Deep Neural Network Model Compression