Abstract:Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can parse natural queries about the visual content and generate human-like outputs. In this work, we explore the ability of these models to demonstrate human-like reasoning based on the perceived information. To address a crucial concern regarding the extent to which their reasoning capabilities are fully consistent and grounded, we also measure the reasoning consistency of these models. We achieve this by proposing a chain-of-thought (CoT) based consistency measure. However, such an evaluation requires a benchmark that encompasses both high-level inference and detailed reasoning chains, which is costly. We tackle this challenge by proposing a LLM-Human-in-the-Loop pipeline, which notably reduces cost while simultaneously ensuring the generation of a high-quality dataset. Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs. We evaluate existing state-of-the-art VLMs, and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency, indicating that substantial efforts are required to enable VLMs to perform visual reasoning as systematically and consistently as humans. As an early step, we propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs. The first stage involves employing supervised fine-tuning of VLMs using step-by-step reasoning samples automatically generated by LLMs. In the second stage, we further augment the training process by incorporating feedback provided by LLMs to produce reasoning chains that are highly consistent and grounded. We empirically highlight the effectiveness of our framework in both reasoning performance and consistency.

Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Enhancing Advanced Visual Reasoning Ability of Large Language Models

Large Language Models are Visual Reasoning Coordinators

LLaVA-o1: Let Vision Language Models Reason Step-by-Step

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models

Enhance Reasoning Ability of Visual-Language Models via Large Language Models

LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

Empowering Vision-Language Models for Reasoning Ability Through Large Language Models

Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models

Enhancing Visual Reasoning with Autonomous Imagination in Multimodal Large Language Models

Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models

Improve Vision Language Model Chain-of-thought Reasoning

ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom

Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets

Multimodal Chain-of-Thought Reasoning in Language Models

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions

Large Language Model with Curriculum Reasoning for Visual Concept Recognition

Reasoning Grasping via Multimodal Large Language Model

Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings