An abundance of datasets and availability of reliable evaluation metrics have
resulted in strong progress in factoid question answering (QA). This progress,
however, does not easily transfer to the task of long-form QA, where the goal
is to answer questions that require in-depth explanations. The hurdles include
(i) a lack of high-quality data, and (ii) the absence of a well-defined notion
of the answer's quality. In this work, we address these problems by (i)
releasing a novel dataset and a task that we call ASQA (Answer Summaries for
Questions which are Ambiguous); and (ii) proposing a reliable metric for
measuring performance on ASQA. Our task focuses on factoid questions that are
ambiguous, that is, have different correct answers depending on interpretation.
Answers to ambiguous questions should synthesize factual information from
multiple sources into a long-form summary that resolves the ambiguity. In
contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear
notion of correctness: a user faced with a good summary should be able to
answer different interpretations of the original ambiguous question. We use
this notion of correctness to define an automated metric of performance for
ASQA. Our analysis demonstrates an agreement between this metric and human
judgments, and reveals a considerable gap between human performance and strong
baselines.

本文提出了一个名为 ASQA 的任务，旨在回答需要深入解释的问题，并提出了一个在 ASQA 上测量表现的可靠度量标准，以及释放了一个新的数据集，用于解决长期问答任务中仍然存在的高质量数据缺乏和答案质量定义模糊的问题。