We present GSM-MC and MATH-MC, two multiple-choice (MC) datasets constructed
by collecting answers and incorrect predictions on GSM8K and MATH from over 50
open-source models. Through extensive experiments, we show that LLMs'
performance on the MC versions of these two popular benchmarks is strongly
correlated with their performance on the original versions, and is quite robust
to distractor choices and option orders, while the evaluation time is reduced
by a factor of up to 30. Following a similar procedure, we also introduce
PythonIO, a new program output prediction MC dataset constructed from two other
popular LLM evaluation benchmarks HumanEval and MBPP. Our data and code are
available at this https URL

我们提出了 GSM-MC 和 MATH-MC 两个多项选择（MC）数据集，通过收集来自 50 多个开源模型对 GSM8K 和 MATH 的答案和错误预测构建。通过广泛的实验，我们展示了 LLMs 在这两个受欢迎基准测试的多项选择版本上的表现与原始版本的表现强烈相关，并且对干扰项选择和选项顺序非常稳健，同时评估时间缩短了多达 30 倍。按照类似的步骤，我们还介绍了 PythonIO，一个由另外两个流行的 LLM 评估基准测试 HumanEval 和 MBPP 构建的新的程序输出预测多项选择数据集。我们的数据和代码可以在此网址找到。