We propose a framework for robust evaluation of reasoning capabilities of
language models, using functional variants of benchmarks. Models that solve a
reasoning test should exhibit no difference in performance over the static
version of a problem compared to a snapshot of the functional variant. We have
rewritten the relevant fragment of the MATH benchmark into its functional
variant MATH(), with functionalization of other benchmarks to follow. When
evaluating current state-of-the-art models over snapshots of MATH(), we find a
reasoning gap -- the percentage difference between the static and functional
accuracies. We find reasoning gaps from 58.35% to 80.31% among the
state-of-the-art closed and open weights models that perform well on static
benchmarks, with the caveat that the gaps are likely to be smaller with more
sophisticated prompting strategies. Here we show that models which anecdotally
have good reasoning performance over real-world tasks, have quantifiable lower
gaps, motivating the open problem of building "gap 0" models. Code for
evaluation and new evaluation datasets, three MATH() snapshots, are publicly
available at this https URL

我们提出了一个用于鲁棒评估语言模型推理能力的框架，使用功能变体基准。在静态问题版本与功能变体快照之间，解决推理测试的模型在性能上不应有差异。我们将 MATH 基准的相关片段重写为其功能变体 MATH ()，其他基准的函数化会随之进行。通过对 MATH () 的快照评估当前最先进的模型，我们发现存在推理差距 - 静态和功能准确性之间的百分比差。我们发现，在静态基准上表现良好的最先进的闭合和开放权重模型中，推理差距从 58.35% 到 80.31% 不等，激励产生 “gap 0” 模型建设的开放问题。评估代码、新的评估数据集以及三个 MATH () 快照可以在此 https URL 处公开获取。