Large Language Models (LLMs) have exhibited remarkable success in long-form
context comprehension tasks. However, their capacity to generate long contents,
such as reports and articles, remains insufficiently explored. Current
benchmarks do not adequately assess LLMs' ability to produce informative and
comprehensive content, necessitating a more rigorous evaluation approach. In
this study, we introduce \textsc{ProxyQA}, a framework for evaluating long-form
text generation, comprising in-depth human-curated \textit{meta-questions}
spanning various domains. Each meta-question contains corresponding
\textit{proxy-questions} with annotated answers. LLMs are prompted to generate
extensive content in response to these meta-questions. Utilizing an evaluator
and incorporating generated content as background context, \textsc{ProxyQA}
evaluates the quality of generated content based on the evaluator's performance
in answering the \textit{proxy-questions}. We examine multiple LLMs,
emphasizing \textsc{ProxyQA}'s demanding nature as a high-quality assessment
tool. Human evaluation demonstrates that evaluating through
\textit{proxy-questions} is a highly self-consistent and
human-criteria-correlated validation method. The dataset and leaderboard will
be available at https://github.com/Namco0816/ProxyQA.

本研究介绍了 ProxyQA 框架，用于评估长文本生成能力，并通过评估器和生成的内容作为背景背景进行生成内容的质量评估，结果表明使用代理问题评估是一种高度自洽且与人工标准相关的验证方法。