Abstract:Pretrained large models, particularly large language models, have garnered increasing attention, as they have demonstrated remarkable abilities through contextual learning. Pretrained large models are increasingly recognized as fundamental tools for solving various tasks. However, the substantial computational demands of large models have dissuaded most product teams and individuals from running them. In such scenarios, to leverage the exceptional performance of large models, one must solely depend on costly APIs, further burdening product teams and individuals. On the other hand, despite the overall inferior performance of small models compared to large models, there are certain distributions where small models can achieve comparable or even superior results. For instance, during training, small models may become trapped in a local optimum that is unique to certain distributions, leading to superior performance. Hence, we propose Data Shunt (DS), a general paradigm for collaboration of small and large models. DS not only substantially reduces the cost associated with deploying large models but also effectively enhances overall performance. Specifically, DS determines the shunting direction by evaluating the confidence level of small models. When the confidence level falls below a specific threshold, the input data is forwarded to large models. To further leverage the advantages of the small and large models, we introduce Prompt Pruning (PP) and 2-Stage Confidence Distillation (2CD), which facilitate mutual collaboration, leading to better results and less cost. The remarkable performance across diverse modalities and tasks demonstrates the superiority of the proposed DS over large models. For instance, ChatGPT achieves an accuracy of 94.43% on Amazon Product sentiment analysis, and DS achieves an accuracy of 95.64%, while the cost has been reduced to only 31.18%. The code for the proposed method are provided for research purposes https://github.com/Anfeather/Data-Shunt.

Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models

Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

One Big Net For Everything

POA: Pre-training Once for Models of All Sizes

Improving Large Models with Small models: Lower Costs and Better Performance

Pre-Trained Models: Past, Present and Future

An Efficient 2D Method for Training Super-Large Deep Learning Models

Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

One-stop Training of Multiple Capacity Models

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Data Shunt: Collaboration of Small and Large Models for Lower Costs and Better Performance

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

12-in-1: Multi-Task Vision and Language Representation Learning

Exploring the Limits of Large Scale Pre-training

GATS: Gather-Attend-Scatter

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Multi-Task Multi-Agent Shared Layers are Universal Cognition of Multi-Agent Coordination

Contrastive Learning for Multiple Models in One Supernet

Massively Multitask Networks for Drug Discovery