This work presents an enhanced approach to generating scene graphs by
incorporating a relationship hierarchy and commonsense knowledge. Specifically,
we propose a Bayesian classification head that exploits an informative
hierarchical structure. It jointly predicts the super-category or type of
relationship between the two objects, along with the detailed relationship
under each super-category. We design a commonsense validation pipeline that
uses a large language model to critique the results from the scene graph
prediction system and then use that feedback to enhance the model performance.
The system requires no external large language model assistance at test time,
making it more convenient for practical applications. Experiments on the Visual
Genome and the OpenImage V6 datasets demonstrate that harnessing hierarchical
relationships enhances the model performance by a large margin. The proposed
Bayesian head can also be incorporated as a portable module in existing scene
graph generation algorithms to improve their results. In addition, the
commonsense validation enables the model to generate an extensive set of
reasonable predictions beyond dataset annotations.

该论文提出了一种改进的方法来生成场景图，通过引入关系层次结构和常识知识。具体而言，我们提出了一个贝叶斯分类头，利用了信息丰富的层次结构，同时预测两个对象之间的超类或关系类型以及每个超类下的详细关系。我们设计了一个常识验证流程，使用大型语言模型对场景图预测系统的结果进行评估，并利用反馈改善模型性能。该系统在测试时不需要外部大型语言模型的辅助，更适用于实际应用。在 Visual Genome 和 OpenImage V6 数据集上的实验证明，利用层次关系可以大幅提高模型性能。该论文提出的贝叶斯头还可以作为一个可移植模块加入现有的场景图生成算法中，以改善它们的结果。此外，常识验证使得模型能够产生大量超出数据集注释的合理预测。

通过层次关系和常识知识增强场景图生成

Enhancing Scene Graph Generation with Hierarchical Relationships and  Commonsense Knowledge

Aleatoric uncertainty captures the inherent randomness of the data, such as
measurement noise. In Bayesian regression, we often use a Gaussian observation
model, where we control the level of aleatoric uncertainty with a noise
variance parameter. By contrast, for Bayesian classification we use a
categorical distribution with no mechanism to represent our beliefs about
aleatoric uncertainty. Our work shows that explicitly accounting for aleatoric
uncertainty significantly improves the performance of Bayesian neural networks.
We note that many standard benchmarks, such as CIFAR, have essentially no
aleatoric uncertainty. Moreover, we show data augmentation in approximate
inference has the effect of softening the likelihood, leading to
underconfidence and profoundly misrepresenting our honest beliefs about
aleatoric uncertainty. Accordingly, we find that a cold posterior, tempered by
a power greater than one, often more honestly reflects our beliefs about
aleatoric uncertainty than no tempering -- providing an explicit link between
data augmentation and cold posteriors. We show that we can match or exceed the
performance of posterior tempering by using a Dirichlet observation model,
where we explicitly control the level of aleatoric uncertainty, without any
need for tempering.

本研究表明，显式地考虑 aleatoric uncertainty 可显著改善贝叶斯神经网络的性能，其中使用 Dirichlet observation model 可以匹配或超过 posterior tempering 的性能，无需 tempering。

关于贝叶斯分类中的不确定性、调和和数据增强

On Uncertainty, Tempering, and Data Augmentation in Bayesian  Classification

In a typical online learning scenario, a learner is required to process a
large data stream using a small memory buffer. Such a requirement is usually in
conflict with a learner's primary pursuit of prediction accuracy. To address
this dilemma, we introduce a novel Bayesian online classi cation algorithm,
called the Virtual Vector Machine. The virtual vector machine allows you to
smoothly trade-off prediction accuracy with memory size. The virtual vector
machine summarizes the information contained in the preceding data stream by a
Gaussian distribution over the classi cation weights plus a constant number of
virtual data points. The virtual data points are designed to add extra
non-Gaussian information about the classi cation weights. To maintain the
constant number of virtual points, the virtual vector machine adds the current
real data point into the virtual point set, merges two most similar virtual
points into a new virtual point or deletes a virtual point that is far from the
decision boundary. The information lost in this process is absorbed into the
Gaussian distribution. The extra information provided by the virtual points
leads to improved predictive accuracy over previous online classification
algorithms.

提出了一种基于贝叶斯定理的虚拟支持向量机算法，通过使用高斯分布和虚拟数据点，平衡在线学习场景下的分类预测精度和内存缓冲的数据处理要求。