Human-object interaction (HOI) detection aims to locate human-object pairs
and identify their interaction categories in images. Most existing methods
primarily focus on supervised learning, which relies on extensive manual HOI
annotations. In this paper, we propose a novel framework, termed Knowledge
Integration to HOI (KI2HOI), that effectively integrates the knowledge of
visual-language model to improve zero-shot HOI detection. Specifically, the
verb feature learning module is designed based on visual semantics, by
employing the verb extraction decoder to convert corresponding verb queries
into interaction-specific category representations. We develop an effective
additive self-attention mechanism to generate more comprehensive visual
representations. Moreover, the innovative interaction representation decoder
effectively extracts informative regions by integrating spatial and visual
feature information through a cross-attention mechanism. To deal with zero-shot
learning in low-data, we leverage a priori knowledge from the CLIP text encoder
to initialize the linear classifier for enhanced interaction understanding.
Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets
demonstrate that our model outperforms the previous methods in various
zero-shot and full-supervised settings.

本研究提出了一种名为 KI2HOI 的新型框架，用于有效整合视觉语言模型的知识，从而改进零样本人物 - 物体交互检测。通过采用动词提取解码器将相应的动词查询转换为特定交互类别的表示形式，设计了基于视觉语义的动词特征学习模块。通过有效的加性自注意机制生成更全面的视觉表示。此外，创新的交互表示解码器通过交叉注意机制，有效地提取信息丰富的区域。为了处理低数据中的零样本学习，利用 CLIP 文本编码器的先验知识，为增强交互理解初始化线性分类器。在主流的 HICO-DET 和 V-COCO 数据集上进行的大量实验证明，我们的模型在各种零样本和全监督设置上优于以前的方法。