As a structured representation of the image content, the visual scene graph
(visual relationship) acts as a bridge between computer vision and natural
language processing. Existing models on the scene graph generation task
notoriously require tens or hundreds of labeled samples. By contrast, human
beings can learn visual relationships from a few or even one example. Inspired
by this, we design a task named One-Shot Scene Graph Generation, where each
relationship triplet (e.g., "dog-has-head") comes from only one labeled
example. The key insight is that rather than learning from scratch, one can
utilize rich prior knowledge. In this paper, we propose Multiple Structured
Knowledge (Relational Knowledge and Commonsense Knowledge) for the one-shot
scene graph generation task. Specifically, the Relational Knowledge represents
the prior knowledge of relationships between entities extracted from the visual
content, e.g., the visual relationships "standing in", "sitting in", and "lying
in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes
"sense-making" knowledge like "dog can guard yard". By organizing these two
kinds of knowledge in a graph structure, Graph Convolution Networks (GCNs) are
used to extract knowledge-embedded semantic features of the entities. Besides,
instead of extracting isolated visual features from each entity generated by
Faster R-CNN, we utilize an Instance Relation Transformer encoder to fully
explore their context information. Based on a constructed one-shot dataset, the
experimental results show that our method significantly outperforms existing
state-of-the-art methods by a large margin. Ablation studies also verify the
effectiveness of the Instance Relation Transformer encoder and the Multiple
Structured Knowledge.

通过使用图形卷积网络和实例关系转换编码器，利用先前的关系和常识知识，提出一种用于一次性场景图生成任务的多重结构化知识方法，并且在构建的数据集上实现了历史最佳的结果。