Abstract:Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting $<human, action, object>$ triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as \emph{\textbf{UniHOI}}. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (\emph{i.e.} GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: \url{<a class="link-external link-https" href="https://github.com/Caoyichao/UniHOI" rel="external noopener nofollow">this https URL</a>}.

UniNL: Aligning Representation Learning with Scoring Function for OOD Detection Via Unified Neighborhood Learning

KNN-Contrastive Learning for Out-of-Domain Intent Classification

Class Lifelong Learning for Intent Detection via Structure Consolidation Networks

'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialogue

Out-of-domain Detection for Natural Language Understanding in Dialog Systems

Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning

MetaOOD: Automatic Selection of OOD Detection Models

Unified Classification and Rejection: A One-versus-All Framework

Learning Object-Language Alignments for Open-Vocabulary Object Detection

Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

Unified Lexical Representation for Interpretable Visual-Language Alignment

Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection

The Open-World Lottery Ticket Hypothesis for OOD Intent Classification

APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection

Uncertainty Aware Learning for Language Model Alignment

Open-Vocabulary Object Detection via Neighboring Region Attention Alignment

Joint Intent Detection and Entity Linking on Spatial Domain Queries.

UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation