The panoptic segmentation task requires a unified result from semantic and
instance segmentation outputs that may contain overlaps. However, current
studies widely ignore modeling overlaps. In this study, we aim to model overlap
relations among instances and resolve them for panoptic segmentation. Inspired
by scene graph representation, we formulate the overlapping problem as a
simplified case, named scene overlap graph. We leverage each object's category,
geometry and appearance features to perform relational embedding, and output a
relation matrix that encodes overlap relations. In order to overcome the lack
of supervision, we introduce a differentiable module to resolve the overlap
between any pair of instances. The mask logits after removing overlaps are fed
into per-pixel instance \verb|id| classification, which leverages the panoptic
supervision to assist in the modeling of overlap relations. Besides, we
generate an approximate ground truth of overlap relations as the weak
supervision, to quantify the accuracy of overlap relations predicted by our
method. Experiments on COCO and Cityscapes demonstrate that our method is able
to accurately predict overlap relations, and outperform the state-of-the-art
performance for panoptic segmentation. Our method also won the Innovation Award
in COCO 2019 challenge.

本研究旨在建立一种基于场景图表示的简化情形，名为场景重合图，来描述实体之间的重叠关系，并利用关系嵌入方法，以减少监督来解决实例之间的重叠关系。最终的试验结果表明，我们提出的方法能够准确地预测重叠关系，在所有测试集上的表现均优于现有最先进的方法，在 COCO 2019 基准测试中获得了创新奖。

SOGNet: 面向全景分割的场景重叠图网络

SOGNet: Scene Overlap Graph Network for Panoptic Segmentation

Objects and their relationships are critical contents for image
understanding. A scene graph provides a structured description that captures
these properties of an image. However, reasoning about the relationships
between objects is very challenging and only a few recent works have attempted
to solve the problem of generating a scene graph from an image. In this paper,
we present a method that improves scene graph generation by explicitly modeling
inter-dependency among the entire object instances. We design a simple and
effective relational embedding module that enables our model to jointly
represent connections among all related objects, rather than focus on an object
in isolation. Our method significantly benefits the main part of the scene
graph generation task: relationship classification. Using it on top of a basic
Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome
benchmark. We further push the performance by introducing global context
encoding module and geometrical layout encoding module. We validate our final
model, LinkNet, through extensive ablation studies, demonstrating its efficacy
in scene graph generation.

本文提出了一种改进场景图生成的方法，通过显式建模整个对象实例的相互依赖关系，设计了一种简单而有效的关系嵌入模块，使我们的模型能够联合表示所有相关对象之间的连接，而不是只关注单个对象。在基本 Faster R-CNN 的基础上使用它，我们的模型在 Visual Genome 基准测试上取得了最先进的结果。通过引入全局上下文编码模块和几何布局编码模块，我们进一步提高了性能。在广泛的消融研究中，我们验证了我们的最终模型 LinkNet 在场景图生成方面的效果。