The effective assessment of the instruction-following ability of large
language models (LLMs) is of paramount importance. A model that cannot adhere
to human instructions might be not able to provide reliable and helpful
responses. In pursuit of this goal, various benchmarks have been constructed to
evaluate the instruction-following capacity of these models. However, these
benchmarks are limited to a single language and are constructed using automated
approaches, which restricts their applicability and the quality of the test
examples they contain. To bridge this gap, we introduce the FollowEval
benchmark in this paper. This benchmark is composed of instances in both
English and Chinese, and all test examples are crafted by human experts.
Furthermore, the FollowEval benchmark is designed to assess LLMs across five
critical dimensions of instruction following: string manipulation, commonsense
reasoning, logical reasoning, spatial reasoning, and response constraints. To
enhance the complexity and present a sufficient challenge, each test example is
designed to evaluate more than one dimension. We have evaluated various LLMs
using the FollowEval benchmark and found that their performance significantly
lags behind that of humans. This highlights the considerable room for
improvement in the instruction-following ability of these models.

这篇论文介绍了 FollowEval 基准测试，通过人工专家设计的测试实例来评估大型语言模型在指令跟随能力方面的表现，测试涵盖了字符串处理、常识推理、逻辑推理、空间推理和响应约束等五个关键维度，并发现这些模型在指令跟随能力方面明显落后于人类，指出了这些模型在这方面还有很大的改进空间。