Abstract:Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZ-HOI achieves state-of-the-art performance across various zero-shot settings with only 10.35% to 33.95% of the trainable parameters compared to existing methods. Code is available at <a class="link-external link-https" href="https://github.com/ChelsieLei/EZ-HOI" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the zero - shot setting, how to effectively detect the Human - Object Interaction (HOI), especially when the model needs to handle unseen classes. Existing methods rely on aligning visual encoders with large visual - language models (VLMs) to utilize the extensive knowledge of VLMs, but this requires a large amount of computational resources and is prone to training difficulties. In addition, directly fine - tuning VLMs may lead to over - fitting to the seen classes, thereby reducing the performance on the unseen classes. To solve these problems, the authors propose a new framework named EZ - HOI, which adapts VLMs through guided prompt learning to achieve efficient zero - shot HOI detection. Specifically, EZ - HOI introduces the following improvements: 1. **Introducing learnable prompts guided by large language models (LLM) and VLM**: - By integrating detailed HOI descriptions and visual semantics, VLMs can better adapt to the HOI task. - Use fixed large language models (LLM) and VLM to guide the prompt learning process, ensuring that the model can understand complex HOI class information. 2. **Unseen Text Prompt Learning (UTPL)**: - Design a new module UTPL to extract information from the related seen classes of the unseen classes. - Use LLM to provide disparity information to enhance the model's learning of the differences and similarities between the unseen and seen classes. 3. **Cross - HOI feature fusion technology**: - Introduce intra - HOI feature fusion and inter - HOI feature fusion to improve the generalization ability and detection accuracy of the model. Through these improvements, EZ - HOI has achieved state - of - the - art performance in the zero - shot HOI detection tasks on multiple benchmark datasets, while significantly reducing the number of trainable parameters and improving training efficiency. ### Formula Summary 1. **Relationship between text prompt and visual prompt**: \[ h_T=\text{Proj}(h_V) \] where \( h_T\in\mathbb{R}^{p\times d_t} \), \( h_V\in\mathbb{R}^{p\times d_v} \), \( p \) is the number of tokens designed for each learnable prompt, \( d_t \) is the dimension of text features, and \( d_v \) is the dimension of visual features. 2. **Update of learnable text prompt under the multi - head cross - attention mechanism**: \[ \hat{h}_T = W_{\text{up}}\cdot\text{MHCA}(Q = W_{\text{down}}\cdot h_T; K, V = W_{\text{down}}\cdot F_{\text{txt}})+h_T \] where \( \hat{h}_T = [\hat{h}_{T1},\hat{h}_{T2},\dots,\hat{h}_{TC}] \), \( \hat{h}_{Ti}\in\mathbb{R}^{p\times d_t} \), \( W_{\text{up}} \) and \( W_{\text{down}} \) represent the up - projection layer and the down - projection layer respectively, and MHCA represents the multi - head cross - attention mechanism. 3. **Refinement of unseen - class text prompt**: \[ \tilde{h}_{Tu}=W_{\text{up}}\cdot\text{MHCA}(Q:W_{\text{down}}\cdot\hat{h}_{Tu}; K,

EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

End-to-End Zero-Shot HOI Detection Via Vision and Language Knowledge Distillation

Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration

CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection.

Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection

Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory

Open-World Human-Object Interaction Detection via Multi-modal Prompts

CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models

Reformulating HOI Detection as Adaptive Set Prediction

Exploring Pose-Aware Human-Object Interaction Via Hybrid Learning

HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model

Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels

Learning Human-Object Interaction via Interactive Semantic Reasoning

Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection

Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models

Improving Human-Object Interaction Detection via Virtual Image Learning