Enhancing the attribution in large language models (LLMs) is a crucial task.
One feasible approach is to enable LLMs to cite external sources that support
their generations. However, existing datasets and evaluation methods in this
domain still exhibit notable limitations. In this work, we formulate the task
of attributed query-focused summarization (AQFS) and present WebCiteS, a
Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS
derives from real-world user queries and web search results, offering a
valuable resource for model training and evaluation. Prior works in attribution
evaluation do not differentiate between groundedness errors and citation
errors. They also fall short in automatically verifying sentences that draw
partial support from multiple sources. We tackle these issues by developing
detailed metrics and enabling the automatic evaluator to decompose the
sentences into sub-claims for fine-grained verification. Our comprehensive
evaluation of both open-source and proprietary models on WebCiteS highlights
the challenge LLMs face in correctly citing sources, underscoring the necessity
for further improvement. The dataset and code will be open-sourced to
facilitate further research in this crucial field.

通过开发详细的度量标准并使自动评估器将句子分解为子主张以进行细粒度验证，我们为获取在正确引用来源方面面临挑战的大型语言模型提出了解决方案，强调了进一步改进的必要性。