Large pre-trained language models have recently been expanded and applied to
programming language tasks with great success, often through further
pre-training of a strictly-natural language model--where training sequences
typically contain both natural and (linearised) programming language. Such
approaches effectively map both modalities of the sequence into the same
embedding space. However, programming language keywords (e.g. ``while'') often
have very strictly defined semantics. As such, transfer learning from their
natural language usage may not necessarily be beneficial to their code
application and vise versa. Assuming an already pre-trained language model, in
this work we investigate how sequence tokens can be adapted and represented
differently, depending on which modality they belong to, and to the ultimate
benefit of the downstream task. We experiment with separating embedding spaces
between modalities during further model pre-training with modality-relative
training objectives. We focus on text-to-code generation and observe consistent
improvements across two backbone models and two test sets, measuring pass@$k$
and a novel incremental variation.

通过改变序列令牌的表示方式，依据所属的模态性进行进一步的模型预训练，并在文本与代码生成任务中观察到一致的改进。

基于模态相对预训练的文本到代码生成

Text-to-Code Generation with Modality-relative Pre-training

Advances in natural language processing, such as transfer learning from
pre-trained language models, have impacted how models are trained for
programming language tasks too. Previous research primarily explored code
pre-training and expanded it through multi-modality and multi-tasking, yet the
data for downstream tasks remain modest in size. Focusing on data utilization
for downstream tasks, we propose and adapt augmentation methods that yield
consistent improvements in code translation and summarization by up to 6.9% and
7.5% respectively. Further analysis suggests that our methods work orthogonally
and show benefits in output code style and numeric consistency. We also discuss
test data imperfections.

本文探讨了自然语言处理、迁移学习和数据利用等技术在编程语言任务中的应用以及基于增强方法的模型训练，该方法可使代码翻译和摘要的准确率分别提高至 6.9% 和 7.5%。