In this paper, we explore a novel task named visual Relation Grounding in Videos (vRGV). The task aims at spatio-temporally localizing the given relations in the form of subject-predicate-object in the videos, so as to provide supportive visual facts for other high-level video-language tasks (e.g., video-language grounding and video question answering). The challenges in this task include but not limited to: (1) both the subject and object are required to be spatio-temporally localized to ground a query relation; (2) the temporal dynamic nature of visual relations in videos is difficult to capture; and (3) the grounding should be achieved without any direct supervision in space and time. To ground the relations, we tackle the challenges by collaboratively optimizing two sequences of regions over a constructed hierarchical spatio-temporal region graph through relation attending and reconstruction, in which we further propose a message passing mechanism by spatial attention shifting between visual entities. Experimental results demonstrate that our model can not only outperform baseline approaches significantly, but also produces visually meaningful facts to support visual grounding. (Code is available at https://github.com/doc-doc/vRGV).

本文介绍了一项新任务：视频中的视觉关系定位，目的是在视频中定位给定的主谓宾形式关系，以提供支持其他高级视频语言任务（例如视频语言基础和视频问答）。 通过协同优化构建的两个区域序列以及关系关注和重构，我们进一步提出了通过视觉实体之间的空间注意力转移的消息传递机制以解决挑战. 我们的模型不仅显着优于基线方法，而且能够产生具有视觉意义的事实以支持视觉基础。