This work presents an enhanced approach to generating scene graphs by incorporating a relationship hierarchy and commonsense knowledge. Specifically, we propose a Bayesian classification head that exploits an informative hierarchical structure. It jointly predicts the super-category or type of relationship between the two objects, along with the detailed relationship under each super-category. We design a commonsense validation pipeline that uses a large language model to critique the results from the scene graph prediction system and then use that feedback to enhance the model performance. The system requires no external large language model assistance at test time, making it more convenient for practical applications. Experiments on the Visual Genome and the OpenImage V6 datasets demonstrate that harnessing hierarchical relationships enhances the model performance by a large margin. The proposed Bayesian head can also be incorporated as a portable module in existing scene graph generation algorithms to improve their results. In addition, the commonsense validation enables the model to generate an extensive set of reasonable predictions beyond dataset annotations.

该论文提出了一种改进的方法来生成场景图，通过引入关系层次结构和常识知识。具体而言，我们提出了一个贝叶斯分类头，利用了信息丰富的层次结构，同时预测两个对象之间的超类或关系类型以及每个超类下的详细关系。我们设计了一个常识验证流程，使用大型语言模型对场景图预测系统的结果进行评估，并利用反馈改善模型性能。该系统在测试时不需要外部大型语言模型的辅助，更适用于实际应用。在Visual Genome和OpenImage V6数据集上的实验证明，利用层次关系可以大幅提高模型性能。该论文提出的贝叶斯头还可以作为一个可移植模块加入现有的场景图生成算法中，以改善它们的结果。此外，常识验证使得模型能够产生大量超出数据集注释的合理预测。

通过层次关系和常识知识增强场景图生成