Despite recent advances, evaluating how well large language models (LLMs)
follow user instructions remains an open problem. While evaluation methods of
language models have seen a rise in prompt-based approaches, limited work on
the correctness of these methods has been conducted. In this work, we perform a
meta-evaluation of a variety of metrics to quantify how accurately they measure
the instruction-following abilities of LLMs. Our investigation is performed on
grounded query-based summarization by collecting a new short-form, real-world
dataset riSum, containing $300$ document-instruction pairs with $3$ answers
each. All $900$ answers are rated by $3$ human annotators. Using riSum, we
analyze agreement between evaluation methods and human judgment. Finally, we
propose new LLM-based reference-free evaluation methods that improve upon
established baselines and perform on-par with costly reference-based metrics
which require high-quality summaries.

通过对大型语言模型的指令遵循能力进行度量的多种度量方法的元评估，分析评估方法与人工判断之间的一致性，并提出基于 LLM 的无参考评估方法，改进了传统基准，并达到了要求高质量摘要的昂贵基于参考文献的指标的效果。