Towards Robust Multi-Modal Reasoning via Model Selection

Xiangyan Liu,Rongxue Li,Wei Ji,Tao Lin

2024-03-23

Abstract:The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the "brain" of the agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on the planning and execution phases, and will only invoke predefined task-specific models for each subtask, making the execution fragile. Meanwhile, other traditional model selection methods are either incompatible with or suboptimal for the multi-modal agent scenarios, due to ignorance of dependencies among subtasks arising by multi-step reasoning. To this end, we identify the key challenges therein and propose the $\textit{M}^3$ framework as a plug-in with negligible runtime overhead at test-time. This framework improves model selection and bolsters the robustness of multi-modal agents in multi-step reasoning. In the absence of suitable benchmarks, we create MS-GQA, a new dataset specifically designed to investigate the model selection challenge in multi-modal agents. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies, thereby robustifying the overall reasoning process. Our code and benchmark:

Artificial Intelligence

What problem does this paper attempt to address?

This paper discusses the importance of model selection in multimodal reasoning. Despite being able to handle complex tasks, current multimodal agents neglect the model selection stage and heavily rely on pre-defined task-specific models, which makes the execution process fragile. The paper proposes the M3 framework, a plug-in with almost no runtime overhead during testing, aiming to improve model selection and enhance the robustness of multimodal agents in multi-step reasoning. Due to the lack of proper benchmarks, the researchers created the MS-GQA dataset specifically for studying model selection challenges in multimodal agents. Experiments show that the M3 framework allows for dynamic model selection, considering user input and subtask dependencies, thereby enhancing the overall reasoning process's stability.

Towards Robust Multi-Modal Reasoning via Model Selection

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models

Progressive Multimodal Reasoning via Active Retrieval

Multi-modal Situated Reasoning in 3D Scenes

Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

Reasoning Grasping via Multimodal Large Language Model

Multimodal Chain-of-Thought Reasoning in Language Models

Multimodal Reasoning with Multimodal Knowledge Graph

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models

ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom

Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning

MaTCR: Modality-Aligned Thought Chain Reasoning for Multimodal Task-Oriented Dialogue Generation

Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive framework

Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agents