Abstract:Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, in which Video-based Human-Object Interaction (V-HOI) detection is a crucial task in semantic scene understanding, aimed at comprehensively understanding HOI relationships within a video to benefit the behavioral decisions of mobile robots and autonomous driving systems. Although previous V-HOI detection models have made significant strides in accurate detection on specific datasets, they still lack the general reasoning ability like human beings to effectively induce HOI relationships. In this study, we propose V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), a novel framework consisting of a series of plug-and-play modules that could facilitate the performance of current V-HOI detection models by leveraging the strong reasoning ability of different off-the-shelf pre-trained large language models (LLMs). We design a two-stage collaboration system of different LLMs for the V-HOI task. Specifically, in the first stage, we design a Cross-Agents Reasoning scheme to leverage the LLM conduct reasoning from different aspects. In the second stage, we perform Multi-LLMs Debate to get the final reasoning answer based on the different knowledge in different LLMs. Additionally, we devise an auxiliary training strategy that utilizes CLIP, a large vision-language model to enhance the base V-HOI models' discriminative ability to better cooperate with LLMs. We validate the superiority of our design by demonstrating its effectiveness in improving the prediction accuracy of the base V-HOI model via reasoning from multiple perspectives.

Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understanding

A User Interface for Sense-making of the Reasoning Process while Interacting with Robots

Towards Metareasoning for Human-Robot Interaction

Multi-Agent Automated Reasoning Toward Machine Self-Awareness: A Case Study.

The Design of Human-Robot Spatial Cognition Interaction System

Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning

Open-Ended Multi-Modal Relational Reasoning for Video Question Answering

Multi-modal Situated Reasoning in 3D Scenes

Closed Loop Interactive Embodied Reasoning for Robot Manipulation

A visual reasoning-based approach for mutual-cognitive human-robot collaboration

Integrated Commonsense Reasoning and Deep Learning for Transparent Decision Making in Robotics

Developing high-level cognitive functions for service robots

Multi-Level Compositional Reasoning for Interactive Instruction Following

Anytime Game-Theoretic Planning with Active Reasoning about Humans' Latent States for Human-Centered Robots

Situational Awareness Matters in 3D Vision Language Reasoning

Multi-modal Human-machine Conversation System for Real Physical World

Simultaneous collaborative mapping and reasoning in dynamic unstructured environments

Multi-Agent Strategy Explanations for Human-Robot Collaboration

Multimodal Reinforcement Learning for Robots Collaborating with Humans

Multi-Robot Collaborative Reasoning for Unique Person Recognition in Complex Environments

Counterfactual Scenario-relevant Knowledge-enriched Multi-modal Emotion Reasoning