Scene graph generation aims to produce structured representations for images,
which requires to understand the relations between objects. Due to the
continuous nature of deep neural networks, the prediction of scene graphs is
divided into object detection and relation classification. However, the
independent relation classes cannot separate the visual features well. Although
some methods organize the visual features into graph structures and use message
passing to learn contextual information, they still suffer from drastic
intra-class variations and unbalanced data distributions. One important factor
is that they learn an unstructured output space that ignores the inherent
structures of scene graphs. Accordingly, in this paper, we propose a Higher
Order Structure Embedded Network (HOSE-Net) to mitigate this issue. First, we
propose a novel structure-aware embedding-to-classifier(SEC) module to
incorporate both local and global structural information of relationships into
the output space. Specifically, a set of context embeddings are learned via
local graph based message passing and then mapped to a global structure based
classification space. Second, since learning too many context-specific
classification subspaces can suffer from data sparsity issues, we propose a
hierarchical semantic aggregation(HSA) module to reduces the number of
subspaces by introducing higher order structural information. HSA is also a
fast and flexible tool to automatically search a semantic object hierarchy
based on relational knowledge graphs. Extensive experiments show that the
proposed HOSE-Net achieves the state-of-the-art performance on two popular
benchmarks of Visual Genome and VRD.

本文提出了一种新的 Higher Order Structure Embedded Network (HOSE-Net) 模型，通过结构感知嵌入到分类器模块和逐层语义聚合模块，用于解决场景图生成中的场景结构问题，并在视觉基因组和 VRD 数据集上取得了最好的性能表现。