Instruction-finetuned code language models (LMs) have shown promise in
various programming tasks. They are trained, using a language modeling
objective, on natural language instructions and gold code snippet pairs. Recent
evidence suggests that these models, never exposed to incorrect solutions
during training, often struggle to distinguish between correct and incorrect
solutions. This observation raises our inquiry: Can preference learning, which
trains models to prefer correct solutions over incorrect ones, help push the
boundaries of code LMs even further? We propose PLUM, a novel
\textbf{p}reference \textbf{l}earning framework a\textbf{u}gmented with test
cases tailored for code L\textbf{M}s.PLUM aims to investigate the key success
factors and potential benefits of preference learning in code LMs, which remain
elusive despite its success in aligning LMs with human values. PLUM consists of
three stages: (1) Generating test cases for natural language instructions, (2)
sampling candidate solutions from the policy and evaluating them against the
test cases to create a preference dataset, which is then used to (3) train the
policy with a preference learning algorithm. Experiments demonstrate that PLUM
substantially improves the performance of existing code LMs on established code
generation benchmarks such as HumanEval (+) and MBPP (+), even for the
state-of-the-art open-source language model CodeQwen-1.5-7B-Chat. PLUM
complements the supervised fine-tuning (SFT) stage, demonstrating synergistic
effects.

PLUM 是一个增加了针对代码 LM 的测试用例的偏好学习框架，它通过三个阶段的实验表明，PLUM 显著提高了现有代码 LM 在代码生成任务中的性能，并与监督微调阶段相互协作产生协同效应。