Abstract:The demand for deploying large language models(LLMs) on mobile devices continues to increase, driven by escalating data security concerns and cloud costs. However, network bandwidth and memory limitations pose challenges for deploying billion-level models on mobile devices. In this study, we investigate the outputs of different layers across various scales of LLMs and found that the outputs of most layers exhibit significant similarity. Moreover, this similarity becomes more pronounced as the model size increases, indicating substantial redundancy in the depth direction of the LLMs. Based on this observation, we propose an efficient model volume compression strategy, termed FoldGPT, which combines block removal and block parameter sharing.This strategy consists of three parts: (1) Based on the learnable gating parameters, we determine the block importance ranking while modeling the coupling effect between blocks. Then we delete some redundant layers based on the given removal rate. (2) For the retained blocks, we apply a specially designed group parameter sharing strategy, where blocks within the same group share identical weights, significantly compressing the number of parameters and slightly reducing latency overhead. (3) After sharing these Blocks, we "cure" the mismatch caused by sparsity with a minor amount of fine-tuning and introduce a tail-layer distillation strategy to improve the performance. Experiments demonstrate that FoldGPT outperforms previous state-of-the-art(SOTA) methods in efficient model compression, demonstrating the feasibility of achieving model lightweighting through straightforward block removal and parameter sharing.

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning

Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning

Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference

SlimGPT: Layer-wise Structured Pruning for Large Language Models

Understanding LLMs: A Comprehensive Overview from Training to Inference

Improving Large Models with Small models: Lower Costs and Better Performance

ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees

TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

Mini-GPTs: Efficient Large Language Models through Contextual Pruning

FoldGPT: Simple and Effective Large Language Model Compression Scheme

FLM-101B: An Open LLM and How to Train It with $100K Budget

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Large Language Models (LLMs): Deployment, Tokenomics and Sustainability

Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level

CPM-2: Large-scale Cost-effective Pre-trained Language Models

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic