Keywords, that is, content-relevant words in summaries play an important role
in efficient information conveyance, making it critical to assess if
system-generated summaries contain such informative words during evaluation.
However, existing evaluation metrics for extreme summarization models do not
pay explicit attention to keywords in summaries, leaving developers ignorant of
their presence. To address this issue, we present a keyword-oriented evaluation
metric, dubbed ROUGE-K, which provides a quantitative answer to the question of
-- \textit{How well do summaries include keywords?} Through the lens of this
keyword-aware metric, we surprisingly find that a current strong baseline model
often misses essential information in their summaries. Our analysis reveals
that human annotators indeed find the summaries with more keywords to be more
relevant to the source documents. This is an important yet previously
overlooked aspect in evaluating summarization systems. Finally, to enhance
keyword inclusion, we propose four approaches for incorporating word importance
into a transformer-based model and experimentally show that it enables guiding
models to include more keywords while keeping the overall quality. Our code is
released at this https URL

我们提出了一种关键词取向的评估指标 ——ROUGE-K，该指标通过定量回答 “摘要中是否包含关键词” 这个问题来评估自动摘要模型是否包含有信息性的单词。通过这种关键词感知的度量方法，我们令人惊讶地发现，当前的一个强基准模型在摘要中常常会遗漏重要信息。我们的分析揭示了人工注释者确实认为包含更多关键词的摘要与源文档更相关。最后，为了增加关键词的包含，我们提出了四种将词重要性融入基于变换器的模型的方法，并通过实验证明这样做可以在保持总体质量的同时引导模型包含更多关键词。