Large language models (LLMs) have demonstrated an impressive ability to
generate code for various programming tasks. In many instances, LLMs can
generate a correct program for a task when given numerous trials. Consequently,
a recent trend is to do large scale sampling of programs using a model and then
filtering/ranking the programs based on the program execution on a small number
of known unit tests to select one candidate solution. However, these approaches
assume that the unit tests are given and assume the ability to safely execute
the generated programs (which can do arbitrary dangerous operations such as
file manipulations). Both of the above assumptions are impractical in
real-world software development. In this paper, we propose CodeRanker, a neural
ranker that can predict the correctness of a sampled program without executing
it. Our CodeRanker is fault-aware i.e., it is trained to predict different
kinds of execution information such as predicting the exact compile/runtime
error type (e.g., an IndexError or a TypeError). We show that CodeRanker can
significantly increase the pass@1 accuracy of various code generation models
(including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets.

该论文提出了一个名为 CodeRanker 的神经排序模型，可以预测采样生成程序的正确性和执行信息，从而提高各种代码生成模型的正确性。