Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models

Wenhao Shi,Zhiqiang Hu,Yi Bin,Junhua Liu,Yang Yang,See-Kiong Ng,Lidong Bing,Roy Ka-Wei Lee

2024-10-08

Abstract:Large language models (LLMs) have demonstrated impressive reasoning capabilities, particularly in textual mathematical problem-solving. However, existing open-source image instruction fine-tuning datasets, containing limited question-answer pairs per image, do not fully exploit visual information to enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs (MLLMs). To bridge this gap, we address the lack of high-quality, diverse multimodal mathematical datasets by collecting 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs, creating the MathV360K dataset, which enhances both the breadth and depth of multimodal mathematical questions. We introduce Math-LLaVA, a LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach significantly improves the multimodal mathematical reasoning capabilities of LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V on MathVista's minitest split, and yielding leading performance on Math-V and MathVerse. Furthermore, Math-LLaVA demonstrates enhanced generalizability, showing substantial improvements on the MMMU benchmark. Our research highlights the importance of dataset diversity and synthesis in advancing MLLMs' mathematical reasoning abilities. The code and data are available at: \url{<a class="link-external link-https" href="https://github.com/HZQ950419/Math-LLaVA" rel="external noopener nofollow">this https URL</a>}.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the issue of data scarcity in the field of multimodal mathematical reasoning, particularly the limitation of existing open-source image instruction fine-tuning datasets in terms of the number of question-answer pairs per image. These datasets have not fully utilized visual information to enhance the mathematical reasoning capabilities of Multimodal Large Language Models (MLLMs). To solve this problem, the authors constructed a dataset named MathV360K, which includes 40,000 high-quality images and their corresponding question-answer pairs selected from 24 existing datasets, and further synthesized 320,000 new question-answer pairs. By fine-tuning the LLaVA-1.5 model using this dataset, a new model named Math-LLaVA was obtained. This model demonstrated significantly improved multimodal mathematical reasoning capabilities in multiple benchmarks, achieving leading results on datasets such as MathVista, Math-V, and MathVerse. Additionally, the performance of Math-LLaVA in the MMMU benchmark also proved its good generalization ability.