Current benchmarks for evaluating neural code models focus on only a small
subset of programming languages, excluding many popular languages such as Go or
Rust. To ameliorate this issue, we present the BabelCode framework for
execution-based evaluation of any benchmark in any language. BabelCode enables
new investigations into the qualitative performance of models' memory, runtime,
and individual test case results. Additionally, we present a new code
translation dataset called Translating Python Programming Puzzles (TP3) from
the Python Programming Puzzles (Schuster et al. 2021) benchmark that involves
translating expert-level python functions to any language. With both BabelCode
and the TP3 benchmark, we investigate if balancing the distributions of 14
languages in a training dataset improves a large language model's performance
on low-resource languages. Training a model on a balanced corpus results in, on
average, 12.34% higher $pass@k$ across all tasks and languages compared to the
baseline. We find that this strategy achieves 66.48% better $pass@k$ on
low-resource languages at the cost of only a 12.94% decrease to high-resource
languages. In our three translation tasks, this strategy yields, on average,
30.77% better low-resource $pass@k$ while having 19.58% worse high-resource
$pass@k$.

为了解决现有编程语言评估标准对一些流行编程语言（如 Go 或 Rust）的限制，该论文提出了一个基于执行的语言无关基准评估框架 BabelCode。同时，该论文还提出了一个新的代码翻译数据集 TP3，用于研究通过平衡 14 种语言在训练数据集中的分布来提高大规模语言模型在低资源语言任务中的性能。训练模型在平衡语料库上平均后，该模型在所有任务和语言上的 $pass@k$ 比基线模型高 12.34％。该策略在低资源语言上的 $pass@k$ 提高了 66.48％，而仅以 12.94％的代价降低了高资源语言的 $pass@k$，具有实际应用意义。