Abstract:Pretrained large models, particularly large language models, have garnered increasing attention, as they have demonstrated remarkable abilities through contextual learning. Pretrained large models are increasingly recognized as fundamental tools for solving various tasks. However, the substantial computational demands of large models have dissuaded most product teams and individuals from running them. In such scenarios, to leverage the exceptional performance of large models, one must solely depend on costly APIs, further burdening product teams and individuals. On the other hand, despite the overall inferior performance of small models compared to large models, there are certain distributions where small models can achieve comparable or even superior results. For instance, during training, small models may become trapped in a local optimum that is unique to certain distributions, leading to superior performance. Hence, we propose Data Shunt (DS), a general paradigm for collaboration of small and large models. DS not only substantially reduces the cost associated with deploying large models but also effectively enhances overall performance. Specifically, DS determines the shunting direction by evaluating the confidence level of small models. When the confidence level falls below a specific threshold, the input data is forwarded to large models. To further leverage the advantages of the small and large models, we introduce Prompt Pruning (PP) and 2-Stage Confidence Distillation (2CD), which facilitate mutual collaboration, leading to better results and less cost. The remarkable performance across diverse modalities and tasks demonstrates the superiority of the proposed DS over large models. For instance, ChatGPT achieves an accuracy of 94.43% on Amazon Product sentiment analysis, and DS achieves an accuracy of 95.64%, while the cost has been reduced to only 31.18%. The code for the proposed method are provided for research purposes https://github.com/Anfeather/Data-Shunt.

Accelerating Deep Learning with Shrinkage and Recall

Deep Neural Network Acceleration with Sparse Prediction Layers

Learning Slimming SSD Through Pruning and Knowledge Distillation

Accelerating Deep Learning Systems Via Critical Set Identification and Model Compression.

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

Double shrinking sparse dimension reduction

Improving deep neural networks for LVCSR using dropout and shrinking structure

Learning Instance-wise Sparsity for Accelerating Deep Models

DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks

Sparsing Deep Neural Network Using Semi-Discrete Matrix Decomposition

A MIC-based acceleration model of Deep Learning

Efficient Deep Learning Inference Based on Model Compression.

LSDDL: Layer-Wise Sparsification for Distributed Deep Learning

Large-Scale Deep Learning Optimizations: A Comprehensive Survey

Data Shunt: Collaboration of Small and Large Models for Lower Costs and Better Performance

DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices

Learning Efficient Convolutional Networks Through Network Slimming.

Accelerating SGD Using Flexible Variance Reduction on Large-Scale Datasets

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Channel Pruning for Accelerating Very Deep Neural Networks

Shrinking-Magnifying Approach: New Exploration For Generalizing Neural Networks