Abstract:Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting $<human, action, object>$ triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as \emph{\textbf{UniHOI}}. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (\emph{i.e.} GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: \url{<a class="link-external link-https" href="https://github.com/Caoyichao/UniHOI" rel="external noopener nofollow">this https URL</a>}.

HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

Interaction is Worth More Explanations: Improving Human-Object Interaction Representation with Propositional Knowledge

Proactive Human-Robot Interaction using Visuo-Lingual Transformers

TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models

A3VLM: Actionable Articulation-Aware Vision Language Model

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models

Multi-modal Interaction with Transformers: Bridging Robots and Human with Natural Language

NMM-HRI: Natural Multi-modal Human-Robot Interaction with Voice and Deictic Posture via Large Language Model

Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

Vision-Language Foundation Models as Effective Robot Imitators

Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics

HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid

Large Language Models as Zero-Shot Human Models for Human-Robot Interaction

VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model