MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning

Somnath Kumar,Yash Gadhia,Tanuja Ganu,Akshay Nambi

2024-05-29

Abstract:Recent advancements in Multi-modal Large Language Models (MLLMs) have significantly improved their performance in tasks combining vision and language. However, challenges persist in detailed multi-modal understanding, comprehension of complex tasks, and reasoning over multi-modal information. This paper introduces MMCTAgent, a novel multi-modal critical thinking agent framework designed to address the inherent limitations of current MLLMs in complex visual reasoning tasks. Inspired by human cognitive processes and critical thinking, MMCTAgent iteratively analyzes multi-modal information, decomposes queries, plans strategies, and dynamically evolves its reasoning. Additionally, MMCTAgent incorporates critical thinking elements such as verification of final answers and self-reflection through a novel approach that defines a vision-based critic and identifies task-specific evaluation criteria, thereby enhancing its decision-making abilities. Through rigorous evaluations across various image and video understanding benchmarks, we demonstrate that MMCTAgent (with and without the critic) outperforms both foundational MLLMs and other tool-augmented pipelines.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the limitations of Multimodal Large Language Models (MLLMs) in complex visual reasoning tasks. Despite recent significant progress in combining visual and textual tasks, challenges remain in detailed multimodal understanding, handling complex tasks, and reasoning across multimodal information. Specifically, current MLLMs perform poorly in spatial understanding, text recognition in images, and spatiotemporal dependencies in long videos, limiting their practical applications. To tackle these issues, the paper proposes a new framework called MMCTAgent, a multimodal critical thinking agent framework designed to improve existing methods through the following approaches: 1. **Iterative Analysis and Strategy Planning**: MMCTAgent adopts a human-like cognitive process to gradually analyze multimodal information, decompose queries, formulate strategies, and dynamically adjust its reasoning process. 2. **Tool Augmentation**: It utilizes external tools to acquire additional information, similar to how humans seek more information to make informed decisions. 3. **Vision-based Critical Component**: A new vision-based critical component is introduced to evaluate evidence and hypotheses, analyze text and multimodal data, ensuring the accuracy of answers and consistency in reasoning. Through rigorous evaluation on various image and video understanding benchmarks, the paper demonstrates that MMCTAgent (with or without the critical component) outperforms baseline MLLMs and other tool-augmented pipelines, achieving significant performance improvements across multiple datasets.

MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges

Cantor: Inspiring Multimodal Chain-of-Thought of MLLM

AgentPS: Agentic Process Supervision for Multi-modal Content Quality Assurance through Multi-round QA

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

MaTCR: Modality-Aligned Thought Chain Reasoning for Multimodal Task-Oriented Dialogue Generation

The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

Enhancing Visual Reasoning with Autonomous Imagination in Multimodal Large Language Models

MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps

Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models

Progressive Multimodal Reasoning via Active Retrieval

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer

Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions

Multimodal Chain-of-Thought Reasoning in Language Models

Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning