On the Performance of Multimodal Language Models

Utsav Garg,Erhan Bas

2023-11-28

Abstract:Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of evaluating and comparing the performance of different multimodal instruction tuning methods across various downstream tasks, including complex reasoning, dialogue, image description, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation studies, the paper reveals key insights into integrating multimodal capabilities into large language models (LLMs), aiming to guide architectural choices and highlight the limitations of current methods in terms of data diversity, authenticity, and factuality. Specifically, the paper focuses on the following aspects:

1. **Effectiveness and Generalization of Multimodal Instruction Tuning**: Investigating the performance of different multimodal instruction tuning methods across various tasks to assess their effectiveness and generalization capabilities.

2. **Impact of Architectural Choices**: Analyzing the impact of different visual encoders, visual heads, and data volumes on model performance, particularly exploring whether larger visual encoders, training visual heads, and adjusting language models can lead to performance improvements.

3. **Necessity of Data Diversity**: Emphasizing the importance of data diversity in enhancing the model's generalization ability, especially when dealing with unseen tasks.

4. **Limitations of Existing Methods**: Highlighting the issues with current methods, such as the lack of diverse multimodal instruction datasets and problems with authenticity and factuality in generated responses.

Through these studies, the paper aims to provide valuable guidance for researchers and practitioners to better utilize multimodal versions of LLMs, particularly in image understanding and generating high-quality responses.