Abstract:Large Language Models (LLMs) have shown significant promise as copilots in various tasks. Local deployment of LLMs on edge devices is necessary when handling privacy-sensitive data or latency-sensitive tasks. The computational constraints of such devices make direct deployment of powerful large-scale LLMs impractical, necessitating the Knowledge Distillation from large-scale models to lightweight models. Lots of work has been done to elicit diversity and quality training examples from LLMs, but little attention has been paid to aligning teacher instructional content based on student preferences, akin to "responsive teaching" in pedagogy. Thus, we propose ARTE, dubbed Aligning TeacheR with StudenT PreferencEs, a framework that aligns the teacher model with student preferences to generate tailored training examples for Knowledge Distillation. Specifically, we elicit draft questions and rationales from the teacher model, then collect student preferences on these questions and rationales using students' performance with in-context learning as a proxy, and finally align the teacher model with student preferences. In the end, we repeat the first step with the aligned teacher model to elicit tailored training examples for the student model on the target task. Extensive experiments on academic benchmarks demonstrate the superiority of ARTE over existing instruction-tuning datasets distilled from powerful LLMs. Moreover, we thoroughly investigate the generalization of ARTE, including the generalization of fine-tuned student models in reasoning ability and the generalization of aligned teacher models to generate tailored training data across tasks and students. In summary, our contributions lie in proposing a novel framework for tailored training example generation, demonstrating its efficacy in experiments, and investigating the generalization of both student & aligned teacher models in ARTE.

What problem does this paper attempt to address?

This paper proposes a solution to the problems encountered when deploying large language models (LLMs) on edge devices. Due to the computational limitations of these devices, directly deploying powerful large models is impractical, so knowledge distillation is necessary to transfer the capabilities of large models to lightweight models. Despite existing work focusing on generating diverse and high-quality training samples from LLMs, little attention has been paid to adjusting the guidance of the teacher model to align with the preferences of the student model, i.e., "responsive teaching" in education. To address this, the paper introduces a new framework called ARTE (Aligning Teacher with Student Preferences), which aligns the preferences of the teacher model and the student model to generate customized training examples for knowledge distillation. The ARTE framework consists of three main steps: 1) knowledge extraction, generating drafts of questions and rationales from the teacher model; 2) preference collection, collecting preferences for the draft questions and rationales using the student model's one-shot context learning performance as a proxy; 3) preference alignment, improving the teaching content by aligning the preferences of the teacher model and the student model through direct preference optimization. Finally, the aligned teacher model is used to regenerate customized training examples for the student model, which are used for supervised fine-tuning. Experiments show that the ARTE method outperforms existing instruction adjustment datasets on academic benchmark tests, and further investigates the generalization of inferential ability and the generalization of the teacher model across tasks and the generation of customized training data by the student model. Overall, the main contribution of the paper is the introduction of a novel framework for generating customized training examples, which has been extensively validated through experiments, and the exploration of the generalization capabilities of student models and aligned teacher models.

Aligning Teacher with Student Preferences for Tailored Training Data Generation

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models

Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios

Customizing Student Networks From Heterogeneous Teachers Via Adaptive Knowledge Amalgamation

Pedagogical Alignment of Large Language Models

Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning

Self-Improving Teacher Cultivates Better Student: Distillation Calibration for Multimodal Large Language Models

Pre-training Distillation for Large Language Models: A Design Space Exploration

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation

TAS: Distilling Arbitrary Teacher and Student via a Hybrid Assistant

Reinforced Multi-Teacher Selection for Knowledge Distillation

TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use

Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model

Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment

CITING: Large Language Models Create Curriculum for Instruction Tuning

Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models

Revisiting Knowledge Distillation for Autoregressive Language Models

One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation

Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights