In this work, we propose a novel approach that predicts the relationships
between various entities in an image in a weakly supervised manner by relying
on image captions and object bounding box annotations as the sole source of
supervision. Our proposed approach uses a top-down attention mechanism to align
entities in captions to objects in the image, and then leverage the syntactic
structure of the captions to align the relations. We use these alignments to
train a relation classification network, thereby obtaining both grounded
captions and dense relationships. We demonstrate the effectiveness of our model
on the Visual Genome dataset by achieving a recall@50 of 15% and recall@100 of
25% on the relationships present in the image. We also show that the model
successfully predicts relations that are not present in the corresponding
captions.

本研究提出一种弱监督学习方法，通过利用图像标题和对象边界框注释作为唯一的监督信息来预测图像中各种实体之间的关系。我们使用自上而下的关注机制将标题中的实体对齐到图像中的对象，然后利用标题的语法结构将关系对齐。我们使用这些对齐来训练关系分类网络，从而获得基于现实的字幕和稠密的关系。我们在 Visual Genome 数据集上展示了我们模型的有效性，其关系的召回率达到 15%（@50）和 25%（@100）。我们还展示了该模型成功预测了与相应字幕中不存在的关系。

从标题和边界框中学习关联

Learning to Relate from Captions and Bounding Boxes

Inspired by recent work in attention models for image captioning and question
answering, we present a soft attention model for the reinforcement learning
domain. This model uses a soft, top-down attention mechanism to create a
bottleneck in the agent, forcing it to focus on task-relevant information by
sequentially querying its view of the environment. The output of the attention
mechanism allows direct observation of the information used by the agent to
select its actions, enabling easier interpretation of this model than of
traditional models. We analyze different strategies that the agents learn and
show that a handful of strategies arise repeatedly across different games. We
also show that the model learns to query separately about space and content
(`where' vs. `what'). We demonstrate that an agent using this mechanism can
achieve performance competitive with state-of-the-art models on ATARI tasks
while still being interpretable.

本文提出了一种结合软性、自上而下的注意机制的强化学习模型，使用信息瓶颈迫使智能体只关注任务相关信息，从而生成容易解释的模型。在 ATARI 游戏上实现了与最先进模型竞争力相当的性能。