Abstract:In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at <a class="link-external link-https" href="https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs" rel="external noopener nofollow">this https URL</a>.

DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMs

Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations

Molecular Graph Representation Learning Integrating Large Language Models with Domain-specific Small Models

General LLMs as Instructors for Domain-Specific LLMs: A Sequential Fusion Method to Integrate Extraction and Editing

DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text

A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks

Efficient Knowledge Infusion via KG-LLM Alignment

DDK: Distilling Domain Knowledge for Efficient Large Language Models

Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs

Knowledge AI: Fine-tuning NLP Models for Facilitating Scientific Knowledge Extraction and Understanding

MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension

SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models

Y-Mol: A Multiscale Biomedical Knowledge-Guided Large Language Model for Drug Development

InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery

Structure-aware Domain Knowledge Injection for Large Language Models

Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources

A Survey on Knowledge Distillation of Large Language Models

Integrating Chemistry Knowledge in Large Language Models via Prompt Engineering

Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks

DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature

Active Deep Kernel Learning of Molecular Functionalities: Realizing Dynamic Structural Embeddings