Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Hao Shao,Shengju Qian,Han Xiao,Guanglu Song,Zhuofan Zong,Letian Wang,Yu Liu,Hongsheng Li

2024-07-08

Abstract:Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are released to foster further research in this direction.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issues of lack of interpretability and accuracy in Multimodal Large Language Models (MLLMs) when handling complex visual inputs. Specifically, the paper focuses on the following aspects:

1. **Interpretability**: Existing MLLMs are often black-box designs, making it difficult to explain their reasoning processes.
2. **High-Resolution Image Processing**: When the input image has a high resolution or the region of interest is small, existing models struggle to accurately extract key information.
3. **Chain-of-Thought (CoT) Capability**: The paper explores how to introduce chain-of-thought into the visual understanding process to improve the model's reasoning ability and interpretability.

To address these challenges, the authors constructed a large-scale Visual Chain-of-Thought dataset (Visual CoT dataset), containing 438,000 question-answer pairs, each annotated with bounding boxes of key regions. Additionally, approximately 98,000 question-answer pairs include detailed reasoning steps. The paper also proposes a multi-turn processing pipeline that can dynamically focus on visual inputs and provide intermediate interpretable reasoning steps.

Through extensive experimental validation, the paper demonstrates the effectiveness of the proposed framework across various tasks and reveals better reasoning strategies. Furthermore, the paper releases the related dataset, benchmarks, and pre-trained models to facilitate further research in this field.