Large language models (LLMs) have emerged as a widely-used tool for
information seeking, but their generated outputs are prone to hallucination. In
this work, we aim to enable LLMs to generate text with citations, improving
their factual correctness and verifiability. Existing work mainly relies on
commercial search engines and human evaluation, making it challenging to
reproduce and compare with different modeling approaches. We propose ALCE, the
first benchmark for Automatic LLMs' Citation Evaluation. ALCE collects a
diverse set of questions and retrieval corpora and requires building end-to-end
systems to retrieve supporting evidence and generate answers with citations. We
build automatic metrics along three dimensions -- fluency, correctness, and
citation quality -- and demonstrate their strong correlation with human
judgements. Our experiments with state-of-the-art LLMs and novel prompting
strategies show that current systems have considerable room for improvements --
for example, on the ELI5 dataset, even the best model has 49% of its
generations lacking complete citation support. Our extensive analyses further
highlight promising future directions, including developing better retrievers,
advancing long-context LLMs, and improving the ability to synthesize
information from multiple sources.

这项研究介绍了 ALCE，一个自动化 LLMs 引用评估的评测基准，通过自动度量三个维度 - 流畅度、正确性和引用质量，强调了更好的检索器、长文本 LLMs 等方向的改善空间。