One of the most difficult tasks in scene understanding is recognizing
interactions between objects in an image. This task is often called visual
relationship detection (VRD). We consider the question of whether, given
auxiliary textual data in addition to the standard visual data used for
training VRD models, VRD performance can be improved. We present a new deep
model that can leverage additional textual data. Our model relies on a shared
text--image representation of subject-verb-object relationships appearing in
the text, and object interactions in images. Our method is the first to enable
recognition of visual relationships missing in the visual training data and
appearing only in the auxiliary text. We test our approach on two different
text sources: text originating in images and text originating in books. We test
and validate our approach using two large-scale recognition tasks: VRD and
Scene Graph Generation. We show a surprising result: Our approach works better
with text originating in books, and outperforms the text originating in images
on the task of unseen relationship recognition. It is comparable to the model
which utilizes text originating in images on the task of seen relationship
recognition.

提出一种融合图像和文本信息用于视觉关系检测和场景图生成的深度模型，通过共同的文本图像表示方法实现文本补充图像数据，结果表明图像中没有的文本信息可以显著提高模型性能