A series of datasets and models have been proposed for summaries generated
for well-formatted documents such as news articles. Dialogue summaries,
however, have been under explored. In this paper, we present the first dataset
with fine-grained factual error annotations named DIASUMFACT. We define
fine-grained factual error detection as a sentence-level multi-label
classification problem, and we evaluate two state-of-the-art (SOTA) models on
our dataset. Both models yield sub-optimal results, with a macro-averaged F1
score of around 0.25 over 6 error classes. We further propose an unsupervised
model ENDERANKER via candidate ranking using pretrained encoder-decoder models.
Our model performs on par with the SOTA models while requiring fewer resources.
These observations confirm the challenges in detecting factual errors from
dialogue summaries, which call for further studies, for which our dataset and
results offer a solid foundation.

本文介绍了第一个 fine-grained 正误注释数据集 DIASUMFACT，以及通过候选排名使用预训练编码器 - 解码器模型提出的无监督模型 ENDERANKER，该模型表现出与 SOTA 模型相当的性能，并且需要更少的资源。这些发现证实了从对话摘要中检测事实错误的挑战，这为进一步研究提供了坚实的基础。