Sequence-to-sequence models have been used to transform erroneous programs
into correct ones when trained with a large enough dataset. Some recent studies
also demonstrated strong empirical evidence that code review (natural language
instruction about suggestive changes in code) can improve the program repair
further. Large language models, trained with Natural Language (NL) and computer
program corpora, have the capacity to contain inherent knowledge of both. In
this study, we investigate if this inherent knowledge of code and NL can be
utilized to improve automated program repair. We applied PLBART and CodeT5, two
state-of-the-art language models that are pre-trained with both Programming
Language (PL) and Natural Language (NL), on two such natural language-based
program repair datasets and found that the pre-trained language models
fine-tuned with datasets containing both code review and subsequent code
changes notably outperform each of the previous models. We observed that the
pre-trained models improve the previously best-reported results by 9.91% on the
Review4Repair dataset and by 24.72% on the dataset by Tufano et al. This
suggests that a pre-trained sequential model has a better understanding of
natural language and can utilize it much better. We performed an ablation study
to assess the contribution of the pre-training mechanism and the model
architecture. We found that pre-training was significantly more important in
the performance gain than the model architecture. The practical application of
using pre-trained transformer models in the context of automated program repair
is still a long way off. However, our study demonstrates the substantial value
of employing pre-trained models, paving the path for future studies to use more
of these.

探究如何利用预训练语言模型对自动程序修复进行改进，研究表明采用 PLBART 和 CodeT5 两种预训练模型 Fine-Tune 后，能在自然语言的程序修复数据集上显著提高性能，其中预训练机制对性能提升至关重要。

基于代码审查的自动化程序修复：预训练 Transformer 模型的表现如何？

Automated Program Repair Based on Code Review: How do Pre-trained  Transformer Models Perform?

Code summarization and generation empower conversion between programming
language (PL) and natural language (NL), while code translation avails the
migration of legacy code from one PL to another. This paper introduces PLBART,
a sequence-to-sequence model capable of performing a broad spectrum of program
and language understanding and generation tasks. PLBART is pre-trained on an
extensive collection of Java and Python functions and associated NL text via
denoising autoencoding. Experiments on code summarization in the English
language, code generation, and code translation in seven programming languages
show that PLBART outperforms or rivals state-of-the-art models. Moreover,
experiments on discriminative tasks, e.g., program repair, clone detection, and
vulnerable code detection, demonstrate PLBART's effectiveness in program
understanding. Furthermore, analysis reveals that PLBART learns program syntax,
style (e.g., identifier naming convention), logical flow (e.g., if block inside
an else block is equivalent to else if block) that are crucial to program
semantics and thus excels even with limited annotations.

PLBART 是一个可进行广泛的程序和语言理解和生成任务的序列到序列模型，已在 Java 和 Python 函数和相关的自然语言文本上进行预训练，实验证明其在代码总结、生成和翻译方面表现优异，同时在程序理解方面也表现出色。