Evaluating the alignment of large language models (LLMs) with user-defined
coding preferences is a challenging endeavour that requires assessing intricate
textual LLMs' outputs. By relying on automated metrics and static analysis
tools, existing benchmarks fail to assess nuances in user instructions and LLM
outputs, highlighting the need for large-scale datasets and benchmarks for LLM
preference alignment. In this paper, we introduce CodeUltraFeedback, a
preference dataset of 10,000 complex instructions to tune and align LLMs to
coding preferences through AI feedback. We generate responses to the
instructions using a pool of 14 diverse LLMs, which we then annotate according
to their alignment with five coding preferences using the LLM-as-a-Judge
approach with GPT-3.5, producing both numerical and textual feedback. We also
present CODAL-Bench, a benchmark for assessing LLM alignment with these coding
preferences. Our results show that CodeLlama-7B-Instruct, aligned through
reinforcement learning from AI feedback (RLAIF) with direct preference
optimization (DPO) using CodeUltraFeedback's AI feedback data, outperforms 34B
LLMs on CODAL-Bench, validating the utility of CodeUltraFeedback for preference
tuning. Furthermore, we show our DPO-aligned CodeLlama model improves
functional correctness on HumanEval+ compared to the unaligned base model.
Therefore, our contributions bridge the gap in preference tuning of LLMs for
code and set the stage for further advancements in model alignment and RLAIF
for code intelligence. Our code and data are available at
this https URL

通过自动化指标和静态分析工具评估大型语言模型与用户定义的编码偏好之间的对齐是一项具有挑战性的任务。本文介绍了 CodeUltraFeedback，这是一个包含 10,000 个复杂指令的偏好数据集，通过 AI 反馈调整和对齐语言模型与编码偏好。我们通过 14 个不同的语言模型对指令生成响应，并使用 LLM 作为评判器的方法对其对齐性进行了标注。我们还提出了 CODAL-Bench，用于评估语言模型与编码偏好对齐的基准。结果表明，通过使用 CodeUltraFeedback 的 AI 反馈数据，采用增强学习和直接优化编码偏好的方法，CodeLlama-7B-Instruct 在 CODAL-Bench 上优于 34B 模型，验证了 CodeUltraFeedback 在偏好调整方面的实用性。此外，我们还展示了经过优化的 CodeLlama 模型相比于未对齐的基础模型在 HumanEval + 上的功能正确性有所改进。因此，我们的贡献弥合了语言模型对编码偏好的调整差距，并为模型对齐和代码智能的进一步发展奠定了基础。