Abstract:Large language models (LLMs) have achieved remarkable advancements in natural language processing. However, the massive scale and computational demands of these models present formidable challenges when considering their practical deployment in resource-constrained environments. While techniques such as chain-of-thought (CoT) distillation have displayed promise in distilling LLMs into small language models (SLMs), there is a risk that distilled SLMs may still inherit flawed reasoning and hallucinations from LLMs. To address these issues, we propose a twofold methodology: First, we introduce a novel method for distilling the self-evaluation capability from LLMs into SLMs, aiming to mitigate the adverse effects of flawed reasoning and hallucinations inherited from LLMs. Second, we advocate for distilling more comprehensive thinking by incorporating multiple distinct CoTs and self-evaluation outputs, to ensure a more thorough and robust knowledge transfer into SLMs. Experiments on three NLP benchmarks demonstrate that our method significantly improves the performance of distilled SLMs, offering a new perspective for developing more effective and efficient SLMs in resource-constrained environments.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the challenges faced when deploying large language models (LLMs) in resource-constrained environments. Despite the significant advancements LLMs have made in natural language processing, their large size and high computational demands limit their practical applications. To tackle these issues, the paper proposes a new methodology that transfers the self-evaluation and comprehensive thinking capabilities of LLMs to small language models (SLMs) through knowledge distillation techniques. Specifically, the paper addresses the following two problems: 1. **Reducing the Impact of Erroneous Reasoning and Hallucinations**: Even during the chain-of-thought (CoT) distillation process, the distilled SLMs may still inherit erroneous reasoning and hallucinations from the LLMs. These errors can degrade the performance of SLMs. The paper proposes a new method to mitigate these negative impacts by distilling the self-evaluation capabilities of LLMs. 2. **Enhancing the Comprehensive Thinking Ability of SLMs**: A single CoT instance may not capture all the different reasoning paths explored by LLMs, thus limiting the richness of the distilled knowledge. The paper proposes combining multiple different CoTs and self-evaluation outputs to enable SLMs to learn more comprehensive thinking abilities. ### Specific Methodology The paper proposes a two-step method: 1. **Distilling Self-Evaluation Capabilities**: By designing self-evaluation prompts, LLMs generate multiple different CoTs and self-evaluation outputs. These self-evaluation outputs are then used to train SLMs, enabling them to distinguish between correct and incorrect reasoning processes. 2. **Distilling Comprehensive Thinking Abilities**: By combining multiple different CoTs and self-evaluation outputs, SLMs can learn a broader range of reasoning paths and more comprehensive thinking abilities. ### Experimental Results Experiments were conducted on three natural language processing benchmark datasets, and the results show that: - The proposed method significantly outperforms baseline methods (standard distillation and CoT distillation) across all tasks and label types. - Notably, when training with five CoTs and their corresponding self-evaluation outputs, the model performance improves significantly. - The method demonstrates good robustness and adaptability across different model sizes and varying numbers of CoTs and self-evaluation outputs. ### Conclusion The proposed method not only improves the performance and reliability of SLMs but also provides a new perspective for developing efficient and effective SLMs in resource-constrained environments.

Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models

Mixed Distillation Helps Smaller Language Models Reason Better

Distilling Mathematical Reasoning Capabilities into Small Language Models

Mixed Distillation Helps Smaller Language Model Better Reasoning

Large Language Models Are Better Reasoners with Self-Verification

Teaching Small Language Models Reasoning Through Counterfactual Distillation

Distilling System 2 into System 1

Large Language Models are reasoners with Self-Verification

Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation

Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning

Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Models

Self-Cognition in Large Language Models: An Exploratory Study

ToM-LM: Delegating Theory of Mind Reasoning to External Symbolic Executors in Large Language Models

Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task

Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication

Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives

Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge