As language models are adapted by a more sophisticated and diverse set of
users, the importance of guaranteeing that they provide factually correct
information supported by verifiable sources is critical across fields of study
& professions. This is especially the case for high-stakes fields, such as
medicine and law, where the risk of propagating false information is high and
can lead to undesirable societal consequences. Previous work studying
factuality and attribution has not focused on analyzing these characteristics
of language model outputs in domain-specific scenarios. In this work, we
present an evaluation study analyzing various axes of factuality and
attribution provided in responses from a few systems, by bringing domain
experts in the loop. Specifically, we first collect expert-curated questions
from 484 participants across 32 fields of study, and then ask the same experts
to evaluate generated responses to their own questions. We also ask experts to
revise answers produced by language models, which leads to ExpertQA, a
high-quality long-form QA dataset with 2177 questions spanning 32 fields, along
with verified answers and attributions for claims in the answers.

验证与归因对于领域特定的语言模型在高风险领域中提供准确信息至关重要，本研究通过领域专家的参与，对领域特定的语言模型生成的回答进行事实性和归因方面的评估研究，并构建了包含 32 个领域，2177 个问题以及验证答案和归因的高质量长型问答数据集 ExpertQA。