Human-Object Interaction (HOI) detection is a challenging computer vision
task that requires visual models to address the complex interactive
relationship between humans and objects and predict HOI triplets. Despite the
challenges posed by the numerous interaction combinations, they also offer
opportunities for multimodal learning of visual texts. In this paper, we
present a systematic and unified framework (RmLR) that enhances HOI detection
by incorporating structured text knowledge. Firstly, we qualitatively and
quantitatively analyze the loss of interaction information in the two-stage HOI
detector and propose a re-mining strategy to generate more comprehensive visual
representation.Secondly, we design more fine-grained sentence- and word-level
alignment and knowledge transfer strategies to effectively address the
many-to-many matching problem between multiple interactions and multiple
texts.These strategies alleviate the matching confusion problem that arises
when multiple interactions occur simultaneously, thereby improving the
effectiveness of the alignment process. Finally, HOI reasoning by visual
features augmented with textual knowledge substantially improves the
understanding of interactions. Experimental results illustrate the
effectiveness of our approach, where state-of-the-art performance is achieved
on public benchmarks. We further analyze the effects of different components of
our approach to provide insights into its efficacy.

通过结构化文本知识，我们提出了一个系统且统一的框架 (RmLR) 来提高人物 - 物体交互检测，通过分析相互作用信息的损失并生成更全面的视觉表示，设计了更精细的句子 - 词级对齐和知识传递策略以有效解决多个交互和多个文本之间的匹配问题，进而提高了对交互的理解。实验结果表明我们的方法在公共基准测试上实现了最先进的性能，并进一步分析了我们方法的各个组成部分对效果的影响，为其功效提供了洞察。