Abstract:Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural language understanding tasks. However, they incur higher inference cost and latency compared to smaller LLMs. Knowledge distillation provides a way to use outputs from these large, capable teacher models to train smaller student models which can be used for inference at lower cost and latency, while retaining comparable accuracy. We investigate the efficacy of distillation using the Llama-3.1-405B-Instruct teacher and the smaller Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct student models. Contributions of this work include (a) We evaluate the generalizability of distillation with the above Llama-3.1 teacher-student pairs across different tasks and datasets (b) We show that using synthetic data during distillation significantly improves the accuracy of 8B and 70B models, and when used with reasoning chains, even matches or surpasses the zero-shot accuracy of 405B model on some datasets (c) We empirically show that distillation enables 8B and 70B models to internalize 405B's reasoning ability by using only standard fine-tuning (without customizing any loss function). This allows cost and latency-efficient student model inference. (d) We show pitfalls in evaluation of distillation, and present task-specific evaluation, including both human and LLM-grading, and ground-truth based traditional accuracy benchmarks. This methodical study brings out the fundamental importance of synthetic data quality in knowledge distillation, and of combining multiple, task-specific ways of accuracy and quality evaluation in assessing the effectiveness of distillation.

Using Knowledge Distillation to improve interpretable models in a retail banking context

Distilling Model Knowledge

From Persistent Homology to Reinforcement Learning with Applications for Retail Banking

Distilling Knowledge from Deep Networks with Applications to Healthcare Domain

Learning Interpretation with Explainable Knowledge Distillation

Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

XtracTree: a Simple and Effective Method for Regulator Validation of Bagging Methods Used in Retail Banking

Enhancing Customer Churn Prediction in the Banking Sector through Hybrid Segmented Models with Model-Agnostic Interpretability Techniques

Knowledge Distillation Performs Partial Variance Reduction

Distilled ChatGPT Topic & Sentiment Modeling with Applications in Finance

Improving Churn Detection in the Banking Sector: A Machine Learning Approach with Probability Calibration Techniques

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

Knowledge Distillation as Semiparametric Inference

Machine Learning for Credit Risk Assessment in Banking: An Overview

Towards Understanding Knowledge Distillation

Revisiting Knowledge Distillation under Distribution Shift

Towards Explainable Artificial Intelligence in Banking and Financial Services

Supervised Machine Learning Techniques: An Overview with Applications to Banking

Knowledge Distillation with a Precise Teacher and Prediction with Abstention

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

Enhancing banking governance: A machine learning-based credit risk classification