Abstract:Vision-language models~(VLMs) have shown remarkable advancements in multimodal reasoning tasks. However, they still often generate inaccurate or irrelevant responses due to issues like hallucinated image understandings or unrefined reasoning paths. To address these challenges, we introduce Critic-V, a novel framework inspired by the Actor-Critic paradigm to boost the reasoning capability of VLMs. This framework decouples the reasoning process and critic process by integrating two independent components: the Reasoner, which generates reasoning paths based on visual and textual inputs, and the Critic, which provides constructive critique to refine these paths. In this approach, the Reasoner generates reasoning responses according to text prompts, which can evolve iteratively as a policy based on feedback from the Critic. This interaction process was theoretically driven by a reinforcement learning framework where the Critic offers natural language critiques instead of scalar rewards, enabling more nuanced feedback to boost the Reasoner's capability on complex reasoning tasks. The Critic model is trained using Direct Preference Optimization (DPO), leveraging a preference dataset of critiques ranked by Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results show that the Critic-V framework significantly outperforms existing methods, including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner and constructive feedback from the preference-optimized Critic enables a more reliable and context-sensitive multimodal reasoning process. Our approach provides a promising solution to enhance the reliability of VLMs, improving their performance in real-world reasoning-heavy multimodal applications such as autonomous driving and embodied intelligence.

EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models

VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI

GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition

Vision Language Models See What You Want but not What You See

GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks

AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering

EgoTaskQA: Understanding Human Tasks in Egocentric Videos

EgoNCE++: Do Egocentric Video-Language Models Really Understand Hand-Object Interactions?

Inferring Human Vision in a Human-Like Way: Key Factors Influencing the Cognitive Processing of Level-1 Visual Perspective-Taking

EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation

Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models

Egocentric Vision Language Planning

Right this way: Can VLMs Guide Us to See More to Answer Questions?

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

MM-Ego: Towards Building Egocentric Multimodal LLMs

Tackling Vision Language Tasks Through Learning Inner Monologues

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors

On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving

Enhancing machine vision: the impact of a novel innovative technology on video question-answering

MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?