In visual relationship detection, human-notated relationships can be regarded
as determinate relationships. However, there are still large amount of
unlabeled data, such as object pairs with less significant relationships or
even with no relationships. We refer to these unlabeled but potentially useful
data as undetermined relationships. Although a vast body of literature exists,
few methods exploit these undetermined relationships for visual relationship
detection.
In this paper, we explore the beneficial effect of undetermined relationships
on visual relationship detection. We propose a novel multi-modal feature based
undetermined relationship learning network (MF-URLN) and achieve great
improvements in relationship detection. In detail, our MF-URLN automatically
generates undetermined relationships by comparing object pairs with
human-notated data according to a designed criterion. Then, the MF-URLN
extracts and fuses features of object pairs from three complementary modals:
visual, spatial, and linguistic modals. Further, the MF-URLN proposes two
correlated subnetworks: one subnetwork decides the determinate confidence, and
the other predicts the relationships. We evaluate the MF-URLN on two datasets:
the Visual Relationship Detection (VRD) and the Visual Genome (VG) datasets.
The experimental results compared with state-of-the-art methods verify the
significant improvements made by the undetermined relationships, e.g., the
top-50 relation detection recall improves from 19.5% to 23.9% on the VRD
dataset.

提出一种基于多模态特征下的未确定关系学习网络模型，可以自动地通过对对象对进行一定的判定给出未确定关系，并在三个模态的基础上提取特征，进而得到关系的确定度和预测结果，通过实验验证该方法在关系检测上的有效性。

探索视觉关系检测中的未确定关系

On Exploring Undetermined Relationships for Visual Relationship  Detection

We present novel method for image-text multi-modal representation learning.
In our knowledge, this work is the first approach of applying adversarial
learning concept to multi-modal learning and not exploiting image-text pair
information to learn multi-modal feature. We only use category information in
contrast with most previous methods using image-text pair information for
multi-modal embedding. In this paper, we show that multi-modal feature can be
achieved without image-text pair information and our method makes more similar
distribution with image and text in multi-modal feature space than other
methods which use image-text pair information. And we show our multi-modal
feature has universal semantic information, even though it was trained for
category prediction. Our model is end-to-end backpropagation, intuitive and
easily extended to other multi-modal learning work.

本文介绍了一种新的图像 - 文本多模态表示学习方法，该方法利用对抗学习概念进行多模态学习，不利用图像 - 文本对信息进行学习，只使用类别信息，其多模态特征的分布与利用图像 - 文本对信息进行学习的其他方法相比更为相似，并且具有通用的语义信息。该模型是端到端反向传播的，在其他多模态学习工作中易于扩展。