Reasoning about complex visual scenes involves perception of entities and
their relations. Scene graphs provide a natural representation for reasoning
tasks, by assigning labels to both entities (nodes) and relations (edges).
Unfortunately, reasoning systems based on SGs are typically trained in a
two-step procedure: First, training a model to predict SGs from images; Then, a
separate model is created to reason based on predicted SGs. In many domains, it
is preferable to train systems jointly in an end-to-end manner, but SGs are not
commonly used as intermediate components in visual reasoning systems because
being discrete and sparse, scene-graph representations are non-differentiable
and difficult to optimize. Here we propose Differentiable Scene Graphs (DSGs),
an image representation that is amenable to differentiable end-to-end
optimization, and requires supervision only from the downstream tasks. DSGs
provide a dense representation for all regions and pairs of regions, and do not
spend modelling capacity on areas of the images that do not contain objects or
relations of interest. We evaluate our model on the challenging task of
identifying referring relationships (RR) in three benchmark datasets, Visual
Genome, VRD and CLEVR. We describe a multi-task objective, and train in an
end-to-end manner supervised by the downstream RR task. Using DSGs as an
intermediate representation leads to new state-of-the-art performance.

该论文介绍了 Differentiable Scene Graphs (DSGs)—— 一种可微的、可端到端优化的图像表示方法，该方法通过利用有标签的下游任务进行监督，对场景图中的实体和关系进行密集编码，从而在视觉推理任务中取得了新的最优表现，特别在指代关系标识方面表现良好。