Abstract:Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting $<human, action, object>$ triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as \emph{\textbf{UniHOI}}. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (\emph{i.e.} GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: \url{<a class="link-external link-https" href="https://github.com/Caoyichao/UniHOI" rel="external noopener nofollow">this https URL</a>}.

Exploring Pose-Aware Human-Object Interaction Via Hybrid Learning

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

Learning Human-Object Interaction via Interactive Semantic Reasoning

Reformulating HOI Detection as Adaptive Set Prediction

Interaction is Worth More Explanations: Improving Human-Object Interaction Representation with Propositional Knowledge

Spatially-Aware Human-Object Interaction Detection with Cross-Modal Enhancement

Learning Human-Object Interaction Detection Using Interaction Points

Learning Transferable Human-Object Interaction Detector with Natural Language Supervision

Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior

Visual Compositional Learning for Human-Object Interaction Detection

Detailed 2D-3D Joint Representation for Human-Object Interaction

Learning to Detect Human-Object Interactions

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

A Human-Object Interaction Detection Method Inspired by Human Body Part Information

Improving Human-Object Interaction Detection via Virtual Image Learning

Human–object Interaction Recognition Based on Interactivity Detection and Multi-Feature Fusion

Hierarchical HOI Detection Framework Augmented by Human Interactive Intention

HOD: Human-Object Decoupling Network for HOI Detection

Human-Object Interaction Detection via Weak Supervision

HODN: Disentangling Human-Object Feature for HOI Detection

Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection