Scene graph generation has emerged as an important problem in computer
vision. While scene graphs provide a grounded representation of objects, their
locations and relations in an image, they do so only at the granularity of
proposal bounding boxes. In this work, we propose the first, to our knowledge,
framework for pixel-level segmentation-grounded scene graph generation. Our
framework is agnostic to the underlying scene graph generation method and
address the lack of segmentation annotations in target scene graph datasets
(e.g., Visual Genome) through transfer and multi-task learning from, and with,
an auxiliary dataset (e.g., MS COCO). Specifically, each target object being
detected is endowed with a segmentation mask, which is expressed as a
lingual-similarity weighted linear combination over categories that have
annotations present in an auxiliary dataset. These inferred masks, along with a
novel Gaussian attention mechanism which grounds the relations at a pixel-level
within the image, allow for improved relation prediction. The entire framework
is end-to-end trainable and is learned in a multi-task manner with both target
and auxiliary datasets.

该论文提出了第一个像素级分割 - 基于场景图生成的框架，并通过辅助数据集的转移学习和多任务学习解决了目标场景图数据集中缺乏分割标注的问题，其中引入了基于语义相似性权重的线性组合来表达目标对象的分割掩模，并引入了新颖的高斯注意机制实现像素级关系预测，并且该框架支持端到端可训练。