Reasoning about the relationships between object pairs in images is a crucial
task for holistic scene understanding. Most of the existing works treat this
task as a pure visual classification task: each type of relationship or phrase
is classified as a relation category based on the extracted visual features.
However, each kind of relationships has a wide variety of object combination
and each pair of objects has diverse interactions. Obtaining sufficient
training samples for all possible relationship categories is difficult and
expensive. In this work, we propose a natural language guided framework to
tackle this problem. We propose to use a generic bi-directional recurrent
neural network to predict the semantic connection between the participating
objects in the relationship from the aspect of natural language. The proposed
simple method achieves the state-of-the-art on the Visual Relationship
Detection (VRD) and Visual Genome datasets, especially when predicting unseen
relationships (e.g. recall improved from 76.42% to 89.79% on VRD zero-shot
testing set).

本研究提出了一种自然语言引导的框架来解决视觉关系检测任务中对象对之间关系分类的问题，并使用双向循环神经网络以及语义连接来预测参与关系的对象之间的语义连接。该方法在 Visual Relationship Detection 和 Visual Genome 数据集上取得了最佳状态，特别是在未预测关系的情况下。

自然语言引导下的视觉关系检测

Natural Language Guided Visual Relationship Detection

This paper introduces a novel approach for modeling visual relations between
pairs of objects. We call relation a triplet of the form (subject, predicate,
object) where the predicate is typically a preposition (eg. 'under', 'in front
of') or a verb ('hold', 'ride') that links a pair of objects (subject, object).
Learning such relations is challenging as the objects have different spatial
configurations and appearances depending on the relation in which they occur.
Another major challenge comes from the difficulty to get annotations,
especially at box-level, for all possible triplets, which makes both learning
and evaluation difficult. The contributions of this paper are threefold. First,
we design strong yet flexible visual features that encode the appearance and
spatial configuration for pairs of objects. Second, we propose a
weakly-supervised discriminative clustering model to learn relations from
image-level labels only. Third we introduce a new challenging dataset of
unusual relations (UnRel) together with an exhaustive annotation, that enables
accurate evaluation of visual relation retrieval. We show experimentally that
our model results in state-of-the-art results on the visual relationship
dataset significantly improving performance on previously unseen relations
(zero-shot learning), and confirm this observation on our newly introduced
UnRel dataset.

本文介绍了一种新的建模视觉关系方法，设计了强大而灵活的视觉特征，提出了一种弱监督判别式聚类模型来从仅具有图像级标签的数据中学习关系，并引入了一个新的具有挑战性的数据集（UnRel）和详尽的注释，用于准备评估视觉关系检索。实验结果表明，本文提出的模型在视觉关系数据集上取得了显著的提高，并在新引入的 UnRel 数据集上验证了该观察结果的有效性。