As language models (LMs) become capable of handling a wide range of tasks,
their evaluation is becoming as challenging as their development. Most
generation benchmarks currently assess LMs using abstract evaluation criteria
like helpfulness and harmlessness, which often lack the flexibility and
granularity of human assessment. Additionally, these benchmarks tend to focus
disproportionately on specific capabilities such as instruction following,
leading to coverage bias. To overcome these limitations, we introduce the
BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate
nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the
BiGGen Bench is its use of instance-specific evaluation criteria, closely
mirroring the nuanced discernment of human evaluation. We apply this benchmark
to assess 103 frontier LMs using five evaluator LMs. Our code, data, and
evaluation results are all publicly available at
this https URL

通过 BiGGen Bench 的引入，对 77 个不同任务中的九种语言模型的生成能力进行了全面评估，并借助实例特定的评估标准来模拟人类评估的微妙辨别。该研究公开提供了代码、数据和评估结果。