FActScore has gained popularity as a metric to estimate the factuality of
long-form texts generated by Large Language Models (LLMs) in English. However,
there has not been any work in studying the behavior of FActScore in other
languages. This paper studies the limitations of each component in the
four-component pipeline of FActScore in the multilingual setting. We introduce
a new dataset for FActScore on texts generated by strong multilingual LLMs. Our
evaluation shows that LLMs exhibit distinct behaviors in both fact extraction
and fact scoring tasks. No LLM produces consistent and reliable FActScore
across languages with varying levels of resources. We also find that the
knowledge source plays an important role in the quality of the estimated
FActScore. Using Wikipedia as the knowledge source may hinder the true
FActScore of long-form text due to its limited coverage in medium- and
low-resource languages. We also incorporate three mitigations to our knowledge
source that ultimately improve FActScore estimation across all languages.

对 FActScore 在线性多语言模型生成的长文本进行研究，发现不同语言中的 LLMs 在事实提取和事实评分任务中表现出不同行为，并且知识源对 FActScore 的估计质量起着重要作用。使用维基百科作为知识源可能会阻碍中小资源语言长文本的真实 FActScore，但通过三种缓解方法，可以提高对所有语言的 FActScore 估计。

多语言 FActScore 分析

An Analysis of Multilingual FActScore

As generated text becomes more commonplace, it is increasingly important to
evaluate how well-supported such text is by external knowledge sources. Many
approaches for evaluating textual support rely on some method for decomposing
text into its individual subclaims which are scored against a trusted
reference. We investigate how various methods of claim decomposition --
especially LLM-based methods -- affect the result of an evaluation approach
such as the recently proposed FActScore, finding that it is sensitive to the
decomposition method used. This sensitivity arises because such metrics
attribute overall textual support to the model that generated the text even
though error can also come from the metric's decomposition step. To measure
decomposition quality, we introduce an adaptation of FActScore, which we call
DecompScore. We then propose an LLM-based approach to generating decompositions
inspired by Bertrand Russell's theory of logical atomism and neo-Davidsonian
semantics and demonstrate its improved decomposition quality over previous
methods.

研究探讨了生成文本的外部知识支持度评估方法，发现该评估方法对于存证方法的选择非常敏感，并提出了一种改进方法来提高分解质量。

深入研究索赔分解

A Closer Look at Claim Decomposition

Evaluating the factuality of long-form text generated by large language
models (LMs) is non-trivial because (1) generations often contain a mixture of
supported and unsupported pieces of information, making binary judgments of
quality inadequate, and (2) human evaluation is time-consuming and costly. In
this paper, we introduce FActScore (Factual precision in Atomicity Score), a
new evaluation that breaks a generation into a series of atomic facts and
computes the percentage of atomic facts supported by a reliable knowledge
source. We conduct an extensive human evaluation to obtain FActScores of people
biographies generated by several state-of-the-art commercial LMs --
InstructGPT, ChatGPT, and the retrieval-augmented PerplexityAI -- and report
new analysis demonstrating the need for such a fine-grained score (e.g.,
ChatGPT only achieves 58%). Since human evaluation is costly, we also introduce
an automated model that estimates FActScore, using retrieval and a strong
language model, with less than a 2% error rate. Finally, we use this automated
metric to evaluate 6,500 generations from a new set of 13 recent LMs that would
have cost $26K if evaluated by humans, with various findings: GPT-4 and ChatGPT
are more factual than public models, and Vicuna and Alpaca are some of the best
public models.

本文介绍了一种评估生成模型生成的长篇文本真实性的新方法 FActScore，其中将生成的文本分解为一系列原子事实，并计算可靠知识来源支持的原子事实的百分比，从而解决了因生成的含有相关与不相关信息的问题难以进行二元评估的问题；同时，提出了一种自动模型，可以使用检索和强语言模型来估计 FActScore，最终发现 ChatGPT 只有 58% 的精度。