Recent years have seen the successful application of large pre-trained models
to code representation learning, resulting in substantial improvements on many
code-related downstream tasks. But there are issues surrounding their
application to SE tasks. First, the majority of the pre-trained models focus on
pre-training only the encoder of the Transformer. For generation tasks that are
addressed using models with the encoder-decoder architecture, however, there is
no reason why the decoder should be left out during pre-training. Second, many
existing pre-trained models, including state-of-the-art models such as
T5-learning, simply reuse the pre-training tasks designed for natural
languages. Moreover, to learn the natural language description of source code
needed eventually for code-related tasks such as code summarization, existing
pre-training tasks require a bilingual corpus composed of source code and the
associated natural language description, which severely limits the amount of
data for pre-training. To this end, we propose SPT-Code, a sequence-to-sequence
pre-trained model for source code. In order to pre-train SPT-Code in a
sequence-to-sequence manner and address the aforementioned weaknesses
associated with existing pre-training tasks, we introduce three pre-training
tasks that are specifically designed to enable SPT-Code to learn knowledge of
source code, the corresponding code structure, as well as a natural language
description of the code without relying on any bilingual corpus, and eventually
exploit these three sources of information when it is applied to downstream
tasks. Experimental results demonstrate that SPT-Code achieves state-of-the-art
performance on five code-related downstream tasks after fine-tuning.

本文介绍了一种新的基于序列到序列（sequence-to-sequence）的编码表示学习预训练模型 SPT-Code，通过三种特定的预训练任务，可以解决现有预训练任务在自然语言编码中存在的问题，该模型可以有效地应用于五个编码相关的下游任务，并取得了最优秀的表现。