Abstract:Multimodal content generation, which leverages visual information to enhance the comprehension of cross-modal understanding, plays a critical role in Multimodal Information Retrieval. With the development of large language models (LLMs), recent research has adopted visual instruction tuning to inject the knowledge of LLMs into downstream multimodal tasks. The high complexity and great demand for resources urge researchers to study e.cient distillation solutions to transfer the knowledge from pre-trained multimodal models (teachers) to more compact student models. However, the instruction tuning for knowledge distillation in multimodal LLMs is resource-intensive and capability-restricted. The comprehension of students is highly reliant on the teacher models. To address this issue, we propose a novel Multimodal Distillation Calibration framework (MmDC). The main idea is to generate high-quality training instances that challenge student models to comprehend and prompt the teacher to calibrate the knowledge transferred to students, ultimately cultivating a better student model in downstream tasks. This framework comprises two stages: (1) multimodal alignment and (2) knowledge distillation calibration. In the.rst stage, parameter-e.cient.ne-tuning is used to enhance feature alignment between di.erent modalities. In the second stage, we develop a calibration strategy to assess the student model's capability and generate high-quality instances to calibrate knowledge distillation from teacher to student. The experiments on diverse datasets show that our framework e.ciently improves the student model's capabilities. Our 7B-size student model, after three iterations of distillation calibration, outperforms the current state-of-the-art LLaVA-13B model on the ScienceQA and LLaVA Test datasets and also exceeds other strong baselines in a zero-shot setting.

TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise

A Teacher Is Worth A Million Instructions

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement

Large Language Models are In-context Teachers for Knowledge Reasoning

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

LLMaAA: Making Large Language Models as Active Annotators

TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use

Pedagogical Alignment of Large Language Models

Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios

Spoken Language Intelligence of Large Language Models for Language Learning

From “giving a Fish” to “teaching to Fish”: Enhancing ITS Inner Loops with Large Language Models

MindLLM: Lightweight Large Language Model Pre-Training, Evaluation and Domain Application

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models

Small LLMs Are Weak Tool Learners: A Multi-LLM Agent

FreeLM: Fine-Tuning-Free Language Model

Language Models as Science Tutors

Self-Improving Teacher Cultivates Better Student: Distillation Calibration for Multimodal Large Language Models

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners

Teaching Language Models to Self-Improve through Interactive Demonstrations