Abstract:Human–Object Interaction (HOI) detection is important to human-centric scene understanding tasks. Existing works tend to assume that the same verb has similar visual characteristics in different HOI categories, an approach that ignores the diverse semantic meanings of the verb. To address this issue, in this paper, we propose a novel Polysemy Deciphering Network (PD-Net) that decodes the visual polysemy of verbs for HOI detection in three distinct ways. First, we refine features for HOI detection to be polysemy-aware through the use of two novel modules: namely, Language Prior-guided Channel Attention (LPCA) and Language Prior-based Feature Augmentation (LPFA). LPCA highlights important elements in human and object appearance features for each HOI category to be identified; moreover, LPFA augments human pose and spatial features for HOI detection using language priors, enabling the verb classifiers to receive language hints that reduce intra-class variation for the same verb. Second, we introduce a novel Polysemy-Aware Modal Fusion module, which guides PD-Net to make decisions based on feature types deemed more important according to the language priors. Third, we propose to relieve the verb polysemy problem through sharing verb classifiers for semantically similar HOI categories. Furthermore, to expedite research on the verb polysemy problem, we build a new benchmark dataset named HOI-VerbPolysemy (HOI-VP), which includes common verbs (predicates) that have diverse semantic meanings in the real world. Finally, through deciphering the visual polysemy of verbs, our approach is demonstrated to outperform state-of-the-art methods by significant margins on the HICO-DET, V-COCO, and HOI-VP databases. Code and data in this paper are available at https://github.com/MuchHair/PD-Net .

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

Spatially-Aware Human-Object Interaction Detection with Cross-Modal Enhancement

Learning Human-Object Interaction via Interactive Semantic Reasoning

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

An Optimization Model for Human-Object Interaction Detection Inspired by Multi-features

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

Interaction is Worth More Explanations: Improving Human-Object Interaction Representation with Propositional Knowledge

Polysemy Deciphering Network for Robust Human–Object Interaction Detection

Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model

An Improved Human-Object Interaction Detection Network

Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection

Spatial-Aware Multi-Level Parsing Network for Human-Object Interaction

Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models

Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory

Exploring Pose-Aware Human-Object Interaction Via Hybrid Learning

Dual-Branch Knowledge Enhancement Network with Vision-Language Model for Human-Object Interaction Detection

Discovering Syntactic Interaction Clues for Human-Object Interaction Detection

Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration

Toward Open-Set Human Object Interaction Detection

Human Object Interaction Detection via Multi-level Conditioned Network

Learning Human-Object Interaction Detection Using Interaction Points