Abstract:Emerging multimodal large language models (MLLMs) exhibit great potential for chart question answering (CQA). Recent efforts primarily focus on scaling up training datasets (i.e., charts, data tables, and question-answer (QA) pairs) through data collection and synthesis. However, our empirical study on existing MLLMs and CQA datasets reveals notable gaps. First, current data collection and synthesis focus on data volume and lack consideration of fine-grained visual encodings and QA tasks, resulting in unbalanced data distribution divergent from practical CQA scenarios. Second, existing work follows the training recipe of the base MLLMs initially designed for natural images, under-exploring the adaptation to unique chart characteristics, such as rich text elements. To fill the gap, we propose a visualization-referenced instruction tuning approach to guide the training dataset enhancement and model development. Specifically, we propose a novel data engine to effectively filter diverse and high-quality data from existing datasets and subsequently refine and augment the data using LLM-based generation techniques to better align with practical QA tasks and visual encodings. Then, to facilitate the adaptation to chart characteristics, we utilize the enriched data to train an MLLM by unfreezing the vision encoder and incorporating a mixture-of-resolution adaptation strategy for enhanced fine-grained recognition. Experimental results validate the effectiveness of our approach. Even with fewer training examples, our model consistently outperforms state-of-the-art CQA models on established benchmarks. We also contribute a dataset split as a benchmark for future research. Source codes and datasets of this paper are available at <a class="link-external link-https" href="https://github.com/zengxingchen/ChartQA-MLLM" rel="external noopener nofollow">this https URL</a>.

ChartAdapter: Large Vision-Language Model for Chart Summarization

ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules

TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning

TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging

Chart Understanding with Large Language Model

ChartLlama: A Multimodal LLM for Chart Understanding and Generation

ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning

Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning

AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks

On Pre-training of Multimodal Language Models Customized for Chart Understanding

ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language

ChartBench: A Benchmark for Complex Visual Reasoning in Charts

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding

ChartifyText: Automated Chart Generation from Data-Involved Texts via LLM

Chart-to-Text: A Large-Scale Benchmark for Chart Summarization

NovaChart: A Large-scale Dataset Towards Chart Understanding and Generation of Multimodal Large Language Models

StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding