Large language models (LLMs) garner significant attention for their
unprecedented performance, leading to an increasing number of researches
evaluating LLMs. However, these evaluation benchmarks are limited to assessing
the instruction-following capabilities, overlooking the fundamental abilities
that emerge during the pre-training stage. Previous subjective evaluation
methods mainly reply on scoring by API models. However, in the absence of
references, large models have shown limited ability to discern subtle
differences. To bridge the gap, we propose F-Eval, a bilingual evaluation
benchmark to evaluate the fundamental abilities, including expression,
commonsense and logic. The tasks in F-Eval include multi-choice objective
tasks, open-ended objective tasks, reference-based subjective tasks and
reference-free subjective tasks. For reference-free subjective tasks, we devise
new evaluation methods, serving as alternatives to scoring by API models. We
conduct evaluations on 13 advanced LLMs. Results show that our evaluation
methods show higher correlation coefficients and larger distinction than other
evaluators. Additionally, we discuss the influence of different model sizes,
dimensions, and normalization methods. We anticipate that F-Eval will
facilitate the study of LLMs' fundamental abilities.

我们提出了 F-Eval，这是一个双语评估基准，旨在评估大型语言模型的基本能力，包括表达能力、常识和逻辑。我们进行了 13 个先进 LLM 的评估，结果显示我们的评估方法与其他评估者相比具有更高的相关系数和更大的区别，并讨论了不同模型大小、维度和归一化方法的影响。我们期望 F-Eval 能促进对 LLMs 基本能力的研究。