Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning

Xingchen Zeng,Haichuan Lin,Yilin Ye,Wei Zeng

2024-08-11

Abstract:Emerging multimodal large language models (MLLMs) exhibit great potential for chart question answering (CQA). Recent efforts primarily focus on scaling up training datasets (i.e., charts, data tables, and question-answer (QA) pairs) through data collection and synthesis. However, our empirical study on existing MLLMs and CQA datasets reveals notable gaps. First, current data collection and synthesis focus on data volume and lack consideration of fine-grained visual encodings and QA tasks, resulting in unbalanced data distribution divergent from practical CQA scenarios. Second, existing work follows the training recipe of the base MLLMs initially designed for natural images, under-exploring the adaptation to unique chart characteristics, such as rich text elements. To fill the gap, we propose a visualization-referenced instruction tuning approach to guide the training dataset enhancement and model development. Specifically, we propose a novel data engine to effectively filter diverse and high-quality data from existing datasets and subsequently refine and augment the data using LLM-based generation techniques to better align with practical QA tasks and visual encodings. Then, to facilitate the adaptation to chart characteristics, we utilize the enriched data to train an MLLM by unfreezing the vision encoder and incorporating a mixture-of-resolution adaptation strategy for enhanced fine-grained recognition. Experimental results validate the effectiveness of our approach. Even with fewer training examples, our model consistently outperforms state-of-the-art CQA models on established benchmarks. We also contribute a dataset split as a benchmark for future research. Source codes and datasets of this paper are available at <a class="link-external link-https" href="https://github.com/zengxingchen/ChartQA-MLLM" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the limitations of Multimodal Large Language Models (MLLM) in the Chart Question Answering (CQA) task. Specifically:

1. **Dataset Bias Issue**: Existing MLLMs, when handling chart question answering, use training datasets that have distribution biases. These datasets often focus on increasing the amount of data while neglecting fine-grained visual encoding and the diversity of task types. This leads to poor performance of the models in real-world applications.

2. **Visual Instruction Optimization**: Researchers have found that current MLLMs lack sufficient understanding of charts, especially when dealing with complex reasoning questions. Therefore, a new method needs to be developed to generate high-quality visual instruction data to enhance the model's understanding of charts.

3. **Model Adaptability Enhancement**: The paper proposes a method based on visual reference instruction tuning. By improving the data engine to generate more diverse chart types and high-quality question-answer pairs, the base model is adjusted to make it more suitable for handling chart-related tasks.

In summary, the main goal of the paper is to improve the performance of MLLMs in the chart question answering task by enhancing the quality of training data and the model architecture, enabling them to better handle real-world application scenarios.