Abstract:Large Language Models (LLMs) face significant challenges at inference time due to their high computational demands. To address this, we present Performance-Guided Knowledge Distillation (PGKD), a cost-effective and high-throughput solution for production text classification applications. PGKD utilizes teacher-student Knowledge Distillation to distill the knowledge of LLMs into smaller, task-specific models. PGKD establishes an active learning routine between the student model and the LLM; the LLM continuously generates new training data leveraging hard-negative mining, student model validation performance, and early-stopping protocols to inform the data generation. By employing a cyclical, performance-aware approach tailored for highly multi-class, sparsely annotated datasets prevalent in industrial text classification, PGKD effectively addresses training challenges and outperforms traditional BERT-base models and other knowledge distillation methods on several multi-class classification datasets. Additionally, cost and latency benchmarking reveals that models fine-tuned with PGKD are up to 130X faster and 25X less expensive than LLMs for inference on the same classification task. While PGKD is showcased for text classification tasks, its versatile framework can be extended to any LLM distillation task, including language generation, making it a powerful tool for optimizing performance across a wide range of AI applications.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the significant challenges faced by large - scale language models (LLMs) during the inference stage due to their high computational requirements. Specifically, the application of LLMs in production environments is limited by high inference latency and cost, especially in scenarios that require strict service - level agreements (SLAs) and scalability. Moreover, many natural language processing (NLP) tasks only require part of the capabilities of LLMs, such as intent detection in text classification, and these tasks can be accomplished by smaller, faster, and more economical pre - trained language models (PLMs). To address these issues, the paper proposes a new method named Performance - Guided Knowledge Distillation (PGKD). PGKD aims to improve the performance of these small - scale models by efficiently transferring the knowledge of LLMs to smaller task - specific models (student models) during the training process, using LLMs as teacher models. This method is particularly suitable for multi - class, sparsely - labeled datasets commonly found in industrial environments. ### Main Problem Summary: 1. **High Inference Cost and Latency**: LLMs require a large amount of computational resources during inference, resulting in high latency and high cost. 2. **Adaptability in Production Environments**: LLMs have difficulty meeting the strict SLA requirements and scalability needs in production environments. 3. **Mismatch of Task Requirements**: Many NLP tasks only require part of the functions of LLMs, rather than all of their complexity. 4. **Multi - class Classification Challenges**: Multi - class classification tasks in industrial applications usually involve a large number of classes, and existing knowledge distillation methods are difficult to effectively cope with this complexity. By introducing PGKD, the paper aims to provide an efficient and economical solution, enabling small - scale models to achieve performance similar to that of large - scale language models while maintaining a lower inference cost.

Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale

Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

Direct Preference Knowledge Distillation for Large Language Models

DDK: Distilling Domain Knowledge for Efficient Large Language Models

MiniLLM: Knowledge Distillation of Large Language Models

Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment

Knowledge Distillation of Black-Box Large Language Models

GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

A Survey on Knowledge Distillation of Large Language Models

Pre-training Distillation for Large Language Models: A Design Space Exploration

Evolving Knowledge Distillation with Large Language Models and Active Learning

Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments

MiniPLM: Knowledge Distillation for Pre-Training Language Models

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies

Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data

GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation

Reinforced Multi-Teacher Selection for Knowledge Distillation