Despite recent advances, evaluating how well large language models (LLMs) follow user instructions remains an open problem. While evaluation methods of language models have seen a rise in prompt-based approaches, limited work on the correctness of these methods has been conducted. In this work, we perform a meta-evaluation of a variety of metrics to quantify how accurately they measure the instruction-following abilities of LLMs. Our investigation is performed on grounded query-based summarization by collecting a new short-form, real-world dataset riSum, containing $300$ document-instruction pairs with $3$ answers each. All $900$ answers are rated by $3$ human annotators. Using riSum, we analyze agreement between evaluation methods and human judgment. Finally, we propose new LLM-based reference-free evaluation methods that improve upon established baselines and perform on-par with costly reference-based metrics which require high-quality summaries.

通过对大型语言模型的指令遵循能力进行度量的多种度量方法的元评估，分析评估方法与人工判断之间的一致性，并提出基于LLM的无参考评估方法，改进了传统基准，并达到了要求高质量摘要的昂贵基于参考文献的指标的效果。

提升指令遵循评估能力的研究：以摘要为例的案例研究