Abstract:Visual language tasks require AI models to comprehend and reason with both visual and textual content. Driven by the power of Large Language Models (LLMs), two prominent methods have emerged: (1) the hybrid integration between LLMs and Vision-Language Models (VLMs), where visual inputs are firstly converted into language descriptions by VLMs, serving as inputs for LLMs to generate final answer(s); (2) visual feature alignment in language space, where visual inputs are encoded as embeddings and projected to LLMs' language space via further supervised fine-tuning. The first approach provides light training costs and interpretability but is hard to be optimized in an end-to-end fashion. The second approach presents decent performance, but feature alignment usually requires large amounts of training data and lacks interpretability. To tackle this dilemma, we propose a novel approach, Inner Monologue Multi-Modal Optimization (IMMO), to solve complex vision language problems by simulating inner monologue processes, a cognitive process in which an individual engages in silent verbal communication with themselves. We enable LLMs and VLMs to interact through natural language conversation and propose to use a two-stage training process to learn how to do the inner monologue (self-asking questions and answering questions). IMMO is evaluated on two popular tasks and the results suggest by emulating the cognitive phenomenon of internal dialogue, our approach can enhance reasoning and explanation abilities, contributing to the more effective fusion of vision and language models. More importantly, instead of using predefined human-crafted monologues, IMMO learns this process within the deep learning models, promising wider applicability to many different AI problems beyond vision language tasks.

Breaking Boundaries Between Linguistics and Artificial Intelligence

A Survey on Vision-Language-Action Models for Embodied AI

Tackling Vision Language Tasks Through Learning Inner Monologues

Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Visual AI and Linguistic Intelligence Through Steerability and Composability

Vision-Language Intelligence: Tasks, Representation Learning, and Large Models

Visual Error Patterns in Multi-Modal AI: A Statistical Approach

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

Beyond robotic speech: mutual benefits to cognitive psychology and artificial intelligence from the study of multimodal communication

Can 3D Vision-Language Models Truly Understand Natural Language?

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models

Enhancing machine vision: the impact of a novel innovative technology on video question-answering