In this work, we propose a novel approach that predicts the relationships
between various entities in an image in a weakly supervised manner by relying
on image captions and object bounding box annotations as the sole source of
supervision. Our proposed approach uses a top-down attention mechanism to align
entities in captions to objects in the image, and then leverage the syntactic
structure of the captions to align the relations. We use these alignments to
train a relation classification network, thereby obtaining both grounded
captions and dense relationships. We demonstrate the effectiveness of our model
on the Visual Genome dataset by achieving a recall@50 of 15% and recall@100 of
25% on the relationships present in the image. We also show that the model
successfully predicts relations that are not present in the corresponding
captions.

本研究提出一种弱监督学习方法，通过利用图像标题和对象边界框注释作为唯一的监督信息来预测图像中各种实体之间的关系。我们使用自上而下的关注机制将标题中的实体对齐到图像中的对象，然后利用标题的语法结构将关系对齐。我们使用这些对齐来训练关系分类网络，从而获得基于现实的字幕和稠密的关系。我们在 Visual Genome 数据集上展示了我们模型的有效性，其关系的召回率达到 15%（@50）和 25%（@100）。我们还展示了该模型成功预测了与相应字幕中不存在的关系。