Abstract:Multimodal content generation, which leverages visual information to enhance the comprehension of cross-modal understanding, plays a critical role in Multimodal Information Retrieval. With the development of large language models (LLMs), recent research has adopted visual instruction tuning to inject the knowledge of LLMs into downstream multimodal tasks. The high complexity and great demand for resources urge researchers to study e.cient distillation solutions to transfer the knowledge from pre-trained multimodal models (teachers) to more compact student models. However, the instruction tuning for knowledge distillation in multimodal LLMs is resource-intensive and capability-restricted. The comprehension of students is highly reliant on the teacher models. To address this issue, we propose a novel Multimodal Distillation Calibration framework (MmDC). The main idea is to generate high-quality training instances that challenge student models to comprehend and prompt the teacher to calibrate the knowledge transferred to students, ultimately cultivating a better student model in downstream tasks. This framework comprises two stages: (1) multimodal alignment and (2) knowledge distillation calibration. In the.rst stage, parameter-e.cient.ne-tuning is used to enhance feature alignment between di.erent modalities. In the second stage, we develop a calibration strategy to assess the student model's capability and generate high-quality instances to calibrate knowledge distillation from teacher to student. The experiments on diverse datasets show that our framework e.ciently improves the student model's capabilities. Our 7B-size student model, after three iterations of distillation calibration, outperforms the current state-of-the-art LLaVA-13B model on the ScienceQA and LLaVA Test datasets and also exceeds other strong baselines in a zero-shot setting.

Multi-teacher Distillation for Multilingual Spelling Correction

PSDSpell: Pre-Training with Self-Distillation Learning for Chinese Spelling Correction

Fast Multi-Task Learning for Query Spelling Correction

AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference

Query spelling correction for multi-language search engines

MISpeller: Multimodal Information Enhancement for Chinese Spelling Correction

Contextual Multilingual Spellchecker for User Queries

Multi-Task Fine-Tuning on BERT Using Spelling Errors Correction for Chinese Text Classification Robustness

A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models

MLSL-Spell: Chinese Spelling Check Based on Multi-Label Annotation

Multi-Teacher Distillation With Single Model for Neural Machine Translation

ABC-Fusion: Adapter-based Bert-level confusion set fusion approach for Chinese spelling correction

Mitigating Catastrophic Forgetting in Multi-domain Chinese Spelling Correction by Multi-stage Knowledge Transfer Framework

Self-Improving Teacher Cultivates Better Student: Distillation Calibration for Multimodal Large Language Models

A large scale ranker-based system for search query spelling correction

Correcting Chinese Spelling Errors with Phonetic Pre-training

Self-Distillation and Pinyin Character Prediction for Chinese Spelling Correction Based on Multimodality

Distilling a Pretrained Language Model to a Multilingual ASR Model

Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction

Multi-Granularity Semantic Revision for Large Language Model Distillation

Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models