The introduction of large language models has significantly advanced code
generation. However, open-source models often lack the execution capabilities
and iterative refinement of advanced systems like the GPT-4 Code Interpreter.
To address this, we introduce OpenCodeInterpreter, a family of open-source code
systems designed for generating, executing, and iteratively refining code.
Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions,
OpenCodeInterpreter integrates execution and human feedback for dynamic code
refinement. Our comprehensive evaluation of OpenCodeInterpreter across key
benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus
reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves
an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and
MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6)
with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap
between open-source code generation models and proprietary systems like GPT-4
Code Interpreter.

OpenCodeInterpreter 是一种开源代码系统，通过集成执行和人类反馈进行动态代码改进，从而弥补了开源代码生成模型与专有系统（如 GPT-4 代码解释器）之间的差距。

OpenCodeInterpreter: 将代码生成与执行和细化相整合

OpenCodeInterpreter: Integrating Code Generation with Execution and  Refinement

The recently released GPT-4 Code Interpreter has demonstrated remarkable
proficiency in solving challenging math problems, primarily attributed to its
ability to seamlessly reason with natural language, generate code, execute
code, and continue reasoning based on the execution output. In this paper, we
present a method to fine-tune open-source language models, enabling them to use
code for modeling and deriving math equations and, consequently, enhancing
their mathematical reasoning abilities. We propose a method of generating novel
and high-quality datasets with math problems and their code-based solutions,
referred to as MathCodeInstruct. Each solution interleaves natural language,
code, and execution results. We also introduce a customized supervised
fine-tuning and inference approach. This approach yields the MathCoder models,
a family of models capable of generating code-based solutions for solving
challenging math problems. Impressively, the MathCoder models achieve
state-of-the-art scores among open-source LLMs on the MATH (45.2%) and GSM8K
(83.9%) datasets, substantially outperforming other open-source alternatives.
Notably, the MathCoder model not only surpasses ChatGPT-3.5 and PaLM-2 on GSM8K
and MATH but also outperforms GPT-4 on the competition-level MATH dataset. The
dataset and models will be released at this https URL

我们提出了一种方法，通过对开源语言模型进行微调，使其能够使用代码进行建模，并推导出数学方程，从而增强其数学推理能力。我们介绍了一种生成包含数学问题和基于代码的解决方案的新颖高质量数据集的方法，称为 MathCodeInstruct。我们还引入了一种定制的有监督微调和推理方法。这种方法产生了 MathCoder 模型，一组能够生成基于代码的解决方案来解决具有挑战性的数学问题的模型。令人印象深刻的是，MathCoder 模型在 MATH（45.2％）和 GSM8K（83.9％）数据集上取得了开源语言模型的最新得分，远远超过其他开源方案。值得注意的是，MathCoder 模型不仅在 GSM8K 和 MATH 上超过了 ChatGPT-3.5 和 PaLM-2，还超过了 GPT-4 在竞争级别的 MATH 数据集上。数据集和模型将在此 URL 发布。

MathCoder：深化数学推理的 LLMs 中无缝代码集成

MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical  Reasoning

Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
brought significant advancements in addressing math reasoning problems. In
particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
shows remarkable performance on challenging math datasets. In this paper, we
explore the effect of code on enhancing LLMs' reasoning capability by
introducing different constraints on the \textit{Code Usage Frequency} of GPT-4
Code Interpreter. We found that its success can be largely attributed to its
powerful skills in generating and executing code, evaluating the output of code
execution, and rectifying its solution when receiving unreasonable outputs.
Based on this insight, we propose a novel and effective prompting method,
explicit \uline{c}ode-based \uline{s}elf-\uline{v}erification~(CSV), to further
boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
use code to self-verify its answers. In instances where the verification state
registers as ``False'', the model shall automatically amend its solution,
analogous to our approach of rectifying errors during a mathematics
examination. Furthermore, we recognize that the states of the verification
result indicate the confidence of a solution, which can improve the
effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
achieve an impressive zero-shot accuracy on MATH dataset \textbf{(53.9\% $\to$
84.3\%)}.

通过对 GPT-4 Code Interpreter 的代码使用频率引入不同约束，本文探讨了代码在增强 LLMs 推理能力方面的效果。基于这种见解，我们提出了一种新颖有效的提示方法，即基于代码的自验证（CSV），进一步提升了 GPT-4 Code Interpreter 的数学推理潜力。使用 GPT-4 Code Interpreter 和 CSV，我们在 MATH 数据集上实现了令人印象深刻的无需训练准确率（53.9％ → 84.3％）。