Abstract:Recently, to comprehensively improve Vision Language Models (VLMs) for Visual Question Answering (VQA), several methods have been proposed to further reinforce the inference capabilities of VLMs to independently tackle VQA tasks rather than some methods that only utilize VLMs as aids to Large Language Models (LLMs). However, these methods ignore the rich common-sense knowledge inside the given VQA image sampled from the real world. Thus, they cannot fully use the powerful VLM for the given VQA question to achieve optimal performance. Attempt to overcome this limitation and inspired by the human top-down reasoning process, i.e., systematically exploring relevant issues to derive a comprehensive answer, this work introduces a novel, explainable multi-agent collaboration framework by leveraging the expansive knowledge of Large Language Models (LLMs) to enhance the capabilities of VLMs themselves. Specifically, our framework comprises three agents, i.e., Responder, Seeker, and Integrator, to collaboratively answer the given VQA question by seeking its relevant issues and generating the final answer in such a top-down reasoning process. The VLM-based Responder agent generates the answer candidates for the question and responds to other relevant issues. The Seeker agent, primarily based on LLM, identifies relevant issues related to the question to inform the Responder agent and constructs a Multi-View Knowledge Base (MVKB) for the given visual scene by leveraging the build-in world knowledge of LLM. The Integrator agent combines knowledge from the Seeker agent and the Responder agent to produce the final VQA answer. Extensive and comprehensive evaluations on diverse VQA datasets with a variety of VLMs demonstrate the superior performance and interpretability of our framework over the baseline method in the zero-shot setting without extra training cost.

InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understanding

VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering

Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

EVLM: An Efficient Vision-Language Model for Visual Understanding

AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making

A Survey on Vision-Language-Action Models for Embodied AI

AP-VLM: Active Perception Enabled by Vision-Language Models

Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

DeepSeek-VL: Towards Real-World Vision-Language Understanding

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

AVA: Towards Autonomous Visualization Agents through Visual Perception‐Driven Decision‐Making

VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework

Tackling Vision Language Tasks Through Learning Inner Monologues

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement