Code Large Language Models (Code LLMs) have excelled at tasks like code
completion but often miss deeper semantics such as execution effects and
dynamic states. This paper aims to bridge the gap between Code LLMs' reliance
on static text data and the need for thorough semantic understanding for
complex tasks like debugging and program repair. We introduce a novel strategy
to train Code LLMs with comprehensive semantics, encompassing high-level
functional descriptions, local execution effects of individual statements, and
overall input/output behavior, thereby linking static code text with dynamic
execution states. We begin by collecting PyX, a clean code corpus of fully
executable samples with functional descriptions and execution tracing. We
propose training Code LLMs to write code and represent and reason about
execution behaviors using natural language, mimicking human verbal debugging.
This approach led to the development of SemCoder, a Code LLM with only 6.7B
parameters, which shows competitive performance with GPT-3.5-turbo on code
generation and execution reasoning tasks. SemCoder achieves 81.1% on HumanEval
(GPT-3.5-turbo: 76.8%) and 54.5% on CRUXEval-I (GPT-3.5-turbo: 50.3%). We also
study the effectiveness of SemCoder's monologue-style execution reasoning
compared to concrete scratchpad reasoning, showing that our approach integrates
semantics from multiple dimensions more smoothly. Finally, we demonstrate the
potential of applying learned semantics to improve Code LLMs' debugging and
self-refining capabilities.

该论文提出了一种新的策略，通过连接静态代码文本和动态执行状态，训练具有全面语义的 Code LLMs，从而填补 Code LLMs 在诸如调试和程序修复等复杂任务中对深层语义的依赖的差距。该方法通过收集 PyX，一个具有可执行样本、功能描述和执行跟踪的干净代码语料库，训练 Code LLMs 使用自然语言编写代码、表示和推理执行行为，从而开发出了仅有 67 亿参数的 SemCoder，该模型在代码生成和执行推理任务上与 GPT-3.5-turbo 表现相当。