Abstract:Multimodal models typically combine a powerful large language model (LLM) with a vision encoder and are then trained on multimodal data via instruction tuning. While this process adapts LLMs to multimodal settings, it remains unclear whether this adaptation compromises their original language reasoning capabilities. In this work, we explore the effects of multimodal instruction tuning on language reasoning performance. We focus on LLaVA, a leading multimodal framework that integrates LLMs such as Vicuna or Mistral with the CLIP vision encoder. We compare the performance of the original LLMs with their multimodal-adapted counterparts across eight language reasoning tasks. Our experiments yield several key insights. First, the impact of multimodal learning varies between Vicuna and Mistral: we observe a degradation in language reasoning for Mistral but improvements for Vicuna across most tasks. Second, while multimodal instruction learning consistently degrades performance on mathematical reasoning tasks (e.g., GSM8K), it enhances performance on commonsense reasoning tasks (e.g., CommonsenseQA). Finally, we demonstrate that a training-free model merging technique can effectively mitigate the language reasoning degradation observed in multimodal-adapted Mistral and even improve performance on visual tasks.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper mainly explores the impact of multimodal instruction tuning on language reasoning ability and proposes a model fusion technique without additional training to alleviate the degradation of language reasoning ability. Specifically, the paper attempts to answer the following questions: 1. **How does multimodal instruction tuning affect language reasoning performance?** - By comparing the performance of the original large - scale language model (LLM) and the model after multimodal tuning (MLLM) on eight language reasoning tasks, the paper finds that the choice of different basic LLMs has different impacts on language reasoning ability. For example, Mistral shows degradation in most tasks, while Vicuna shows improvement in some tasks. 2. **How does the choice of different basic LLMs affect language reasoning degradation?** - Research shows that choosing different basic LLMs (such as Mistral and Vicuna) will lead to different degrees of language reasoning degradation. A more powerful LLM (such as Mistral) may experience greater degradation after visual instruction tuning. 3. **How to effectively alleviate language reasoning degradation?** - The paper proposes a model fusion method without additional training. By fusing the parameters of the original LLM with the parameters of the multimodal - tuned model, it effectively alleviates the degradation of language reasoning ability. Experimental results show that this fusion method can not only restore language reasoning performance, but also improve the performance of visual tasks. ### Main research findings - **Task - dependence**: The impact of multimodal instruction tuning on language reasoning performance is not uniform. Mathematical reasoning tasks (such as GSM8k) usually degrade, while common - sense reasoning tasks (such as CommonsenseQA) may be improved. - **Importance of model selection**: The choice of different basic LLMs has a significant impact on language reasoning degradation. A stronger LLM (such as Mistral) is more likely to degrade. - **Effectiveness of model fusion**: By adjusting the fusion weight, the balance between language reasoning and visual reasoning can be optimized without additional training, thus effectively alleviating language reasoning degradation. ### Conclusion This research shows that multimodal instruction tuning may lead to the degradation of language reasoning performance, but this degradation can be effectively alleviated by a simple model fusion technique. This provides a new direction for future research, especially in how to optimize the comprehensive performance of multimodal models. ### Formula presentation Some formulas involved in the paper include: - **Task vector calculation**: \[ T=\theta_{\text{vlm}}-\theta_{\text{llm}} \] where \(\theta_{\text{vlm}}\) is the model parameter after multimodal tuning, and \(\theta_{\text{llm}}\) is the parameter of the original LLM. - **Fused model parameter**: \[ M = \alpha\hat{T}+\theta_{\text{vlm}} \] where \(\alpha\) is the fusion weight, and \(\hat{T}\) is the pruned task vector. These formulas show how to alleviate language reasoning degradation through task vector calculation and fusion.

Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

On the Performance of Multimodal Language Models

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models

Diving into Self-Evolving Training for Multimodal Reasoning

The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks

Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

Towards Multimodal In-Context Learning for Vision & Language Models

MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models

MALT: Improving Reasoning with Multi-Agent LLM Training

Acquiring Linguistic Knowledge from Multimodal Input

Visually-Augmented Language Modeling

Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Is Multimodal Vision Supervision Beneficial to Language?

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

MPT4LM: Multi-Modal Prompt Tuning Makes Pre-Trained Large Language Models Better Vision-Language Learners

Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning