Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

Mustafa Dogan,Ilker Kesen,Iacer Calixto,Aykut Erdem,Erkut Erdem

2024-07-17

Abstract:The linguistic capabilities of Multimodal Large Language Models (MLLMs) are critical for their effective application across diverse tasks. This study aims to evaluate the performance of MLLMs on the VALSE benchmark, focusing on the efficacy of few-shot In-Context Learning (ICL), and Chain-of-Thought (CoT) prompting. We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets. The experimental results reveal that ICL and CoT prompting significantly boost model performance, particularly in tasks requiring complex reasoning and contextual understanding. Models pretrained on captioning datasets show superior zero-shot performance, while those trained on interleaved image-text data benefit from few-shot learning. Our findings provide valuable insights into optimizing MLLMs for better grounding of language in visual contexts, highlighting the importance of the composition of pretraining data and the potential of few-shot learning strategies to improve the reasoning abilities of MLLMs.

Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is the evaluation of the language capabilities of Multimodal Large Language Models (MLLMs) under few-shot learning and Chain-of-Thought (CoT) prompting strategies, particularly in the context of visual and language tasks. Specifically, the researchers aim to address this issue through the following points:

1. **Evaluate Model Performance**: Assess the performance of MLLMs of different sizes and pre-training datasets on the VALSE benchmark in zero-shot and few-shot settings.

2. **Explore Strategy Effects**: Investigate the impact of using In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting in few-shot learning on model performance, especially how these strategies enhance the model's performance in complex reasoning and contextual understanding tasks.

3. **Analyze the Impact of Pre-training Data**: Examine the influence of different pre-training datasets (such as image-text pair datasets and subtitle datasets) on the model's zero-shot and few-shot learning capabilities.

4. **Optimize Model Application**: Provide valuable insights for optimizing the application of MLLMs in visual and language tasks through experimental results, particularly how to improve the model's reasoning ability by adjusting the composition of pre-training data and adopting few-shot learning strategies.

In summary, the main objective of this paper is to reveal the potential of ICL and CoT strategies in enhancing the language capabilities of MLLMs through systematic evaluation and analysis, and to provide guidance for future research and practical applications.