Abstract:Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, in which Video-based Human-Object Interaction (V-HOI) detection is a crucial task in semantic scene understanding, aimed at comprehensively understanding HOI relationships within a video to benefit the behavioral decisions of mobile robots and autonomous driving systems. Although previous V-HOI detection models have made significant strides in accurate detection on specific datasets, they still lack the general reasoning ability like human beings to effectively induce HOI relationships. In this study, we propose V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), a novel framework consisting of a series of plug-and-play modules that could facilitate the performance of current V-HOI detection models by leveraging the strong reasoning ability of different off-the-shelf pre-trained large language models (LLMs). We design a two-stage collaboration system of different LLMs for the V-HOI task. Specifically, in the first stage, we design a Cross-Agents Reasoning scheme to leverage the LLM conduct reasoning from different aspects. In the second stage, we perform Multi-LLMs Debate to get the final reasoning answer based on the different knowledge in different LLMs. Additionally, we devise an auxiliary training strategy that utilizes CLIP, a large vision-language model to enhance the base V-HOI models' discriminative ability to better cooperate with LLMs. We validate the superiority of our design by demonstrating its effectiveness in improving the prediction accuracy of the base V-HOI model via reasoning from multiple perspectives.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to enhance the capability of Video Human-Object Interaction (V-HOI) detection through collaborative reasoning with multiple large language models (LLMs). Specifically, the paper addresses the following issues: 1. **Improving the General Reasoning Ability of V-HOI Detection**: - Current V-HOI detection models, although achieving good results on specific datasets, lack the general reasoning ability akin to humans and cannot effectively infer the interaction relationships between humans and objects. For example, in Figure 1, the current state-of-the-art V-HOI model predicts "a person is hugging a fruit," which does not align with common sense; the correct prediction should be "a person is holding a fruit." 2. **Leveraging the Reasoning Ability of LLMs**: - Large language models (LLMs) possess rich common-sense knowledge and strong reasoning abilities. The paper proposes a new framework, V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), to improve the performance of V-HOI detection models through collaborative reasoning with multiple pre-trained large language models. 3. **Designing a Two-Stage Collaborative System**: - The first stage is Cross-Agents Reasoning, where different LLMs reason from different perspectives; - The second stage is Multi-LLMs Debate, where the final answer is derived by integrating the responses from different LLMs. 4. **Introducing Auxiliary Training Strategies**: - Utilizing the CLIP model as internal supervision to enhance the reasoning and differentiation capabilities of the base V-HOI model, better handling the ambiguity of semantic relationships. ### Main Contributions 1. **First Exploration of Using Multiple LLMs to Assist V-HOI Tasks**. 2. **Proposed a Plugin Framework V-HOI MLCR**, which can utilize multiple large language models as external experts to enhance the reasoning ability of current V-HOI models. 3. **Conducted Extensive Experimental Evaluations**, validating the effectiveness of V-HOI MLCR on two V-HOI detection datasets. ### Experimental Results The paper conducted experiments on the Action Genome (AG) dataset and the VidHOI dataset, demonstrating significant improvements in metrics such as Recall@10, Recall@20, and Recall@50. Even with only the common-sense reasoning agent and debate scheme, the V-HOI MLCR framework significantly outperformed the baseline model on the AG dataset. In the more complex multi-person scenario VidHOI dataset, the MLCR framework still performed excellently.

Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom

Multi-modal Large Language Model Enhanced Pseudo 3D Perception Framework for Visual Commonsense Reasoning

Reasoning Grasping via Multimodal Large Language Model

Human-Centric Autonomous Systems with LLMs for User Command Reasoning

Enhancing Advanced Visual Reasoning Ability of Large Language Models

Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection

Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning

Enhancing Visual Reasoning with Autonomous Imagination in Multimodal Large Language Models

OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning

World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

Toward Cognitive Digital Twin System of Human–Robot Collaboration Manipulation

LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

Inquire, Interact, and Integrate: A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoning