We develop a generative attention-based approach to modeling structured
entities comprising different property types, such as numerical, categorical,
string, and composite. This approach handles such heterogeneous data through a
mixed continuous-discrete diffusion process over the properties. Our flexible
framework can model entities with arbitrary hierarchical properties, enabling
applications to structured Knowledge Base (KB) entities and tabular data. Our
approach obtains state-of-the-art performance on a majority of cases across 15
datasets. In addition, experiments with a device KB and a nuclear physics
dataset demonstrate the model's ability to learn representations useful for
entity completion in diverse settings. This has many downstream use cases,
including modeling numerical properties with high accuracy - critical for
science applications, which also benefit from the model's inherent
probabilistic nature.

我们开发了一种生成式基于注意力的方法，用于对结构化实体进行建模，这些实体包括不同类型的属性，如数值、分类、字符串和组合。我们的方法可以处理这种异构数据，并通过连续 - 离散的扩散过程来对属性进行建模。我们的灵活框架可以对具有任意层次属性的实体进行建模，从而能够应用于结构化知识库实体和表格数据。我们的方法在 15 个数据集中的大多数情况下都获得了最先进的性能。此外，通过设备知识库和核物理数据集的实验，证明了该模型在各种环境中学习有用的实体补全表示的能力。这对模型的固有概率特性来说非常重要，在科学应用中对数值属性进行高准确性建模也有很大好处。

KBFormer：用于结构化实体补全的扩散模型

KBFormer: A Diffusion Model for Structured Entity Completion

Although large neural language models (LMs) like BERT can be finetuned to
yield state-of-the-art results on many NLP tasks, it is often unclear what
these models actually learn. Here we study using such LMs to fill in entities
in human-authored comparative questions, like ``Which country is older, India
or ______?'' -- i.e., we study the ability of neural LMs to ask (not answer)
reasonable questions. We show that accuracy in this fill-in-the-blank task is
well-correlated with human judgements of whether a question is reasonable, and
that these models can be trained to achieve nearly human-level performance in
completing comparative questions in three different subdomains. However,
analysis shows that what they learn fails to model any sort of broad notion of
which entities are semantically comparable or similar -- instead the trained
models are very domain-specific, and performance is highly correlated with
co-occurrences between specific entities observed in the training set. This is
true both for models that are pretrained on general text corpora, as well as
models trained on a large corpus of comparison questions. Our study thus
reinforces recent results on the difficulty of making claims about a deep
model's world knowledge or linguistic competence based on performance on
specific benchmark problems. We make our evaluation datasets publicly available
to foster future research on complex understanding and reasoning in such models
at standards of human interaction.

本论文研究以大型神经语言模型填充人类编写的比较问题中的实体，证明在三个不同子域中这些模型的性能可以接近于人类的完成水平，但分析表明它们学习的内容并不能建模任何广泛的可比较或类似的实体，而是非常特定于特定领域，性能与训练集中特定实体的共现高度相关。