Advances in large language models have notably enhanced the efficiency of
information extraction from unstructured and semi-structured data sources. As
these technologies become integral to various applications, establishing an
objective measure for the quality of information extraction becomes imperative.
However, the scarcity of labeled data presents significant challenges to this
endeavor. In this paper, we introduce an automatic framework to assess the
quality of the information extraction and its completeness. The framework
focuses on information extraction in the form of entity and its properties. We
discuss how to handle the input/output size limitations of the large language
models and analyze their performance when iteratively extracting the
information. Finally, we introduce metrics to evaluate the quality of the
extraction and provide an extensive discussion on how to interpret the metrics.

大规模语言模型的进展显著提高了从非结构化和半结构化数据源中提取信息的效率。本文引入了一个自动框架，用于评估信息提取的质量和完整性，重点关注实体及其属性的信息提取。讨论了如何处理大规模语言模型的输入 / 输出大小限制，并分析了在迭代提取信息时它们的性能。最后，介绍了用于评估提取质量的度量标准，并对如何解释这些度量标准进行了广泛讨论。

评估信息提取的质量

Assessing the quality of information extraction

Distributional data tells us that a man can swallow candy, but not that a man
can swallow a paintball, since this is never attested. However both are
physically plausible events. This paper introduces the task of semantic
plausibility: recognizing plausible but possibly novel events. We present a new
crowdsourced dataset of semantic plausibility judgments of single events such
as "man swallow paintball". Simple models based on distributional
representations perform poorly on this task, despite doing well on selection
preference, but injecting manually elicited knowledge about entity properties
provides a substantial performance boost. Our error analysis shows that our new
dataset is a great testbed for semantic plausibility models: more sophisticated
knowledge representation and propagation could address many of the remaining
errors.

本文引入了 “语义可信度” 任务，旨在识别可能是新事件的可信度， 提供了一个新的众包数据集，旨在对单个事件进行语义可信度判断，并发现基于分布式表示的简单模型在此任务上表现不佳，但注入手动引出的实体属性知识可以大幅提高性能，我们的错误分析表明，本文提出的新数据集是进行语义可信度模型测试的极佳试验平台：更复杂的知识表示和传播方法可以解决许多剩余错误。

注入世界知识的语义可信度建模

Modeling Semantic Plausibility by Injecting World Knowledge

The Internet has enabled the creation of a growing number of large-scale
knowledge bases in a variety of domains containing complementary information.
Tools for automatically aligning these knowledge bases would make it possible
to unify many sources of structured knowledge and answer complex queries.
However, the efficient alignment of large-scale knowledge bases still poses a
considerable challenge. Here, we present Simple Greedy Matching (SiGMa), a
simple algorithm for aligning knowledge bases with millions of entities and
facts. SiGMa is an iterative propagation algorithm which leverages both the
structural information from the relationship graph as well as flexible
similarity measures between entity properties in a greedy local search, thus
making it scalable. Despite its greedy nature, our experiments indicate that
SiGMa can efficiently match some of the world's largest knowledge bases with
high precision. We provide additional experiments on benchmark datasets which
demonstrate that SiGMa can outperform state-of-the-art approaches both in
accuracy and efficiency.

本文介绍了一个名为 SiGMa 的算法，它是一种对齐具有数百万实体和事实的知识库的简单且高效的算法，并且可以在准确性和效率方面优于现有方法。