Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.

本文介绍了一种新的预训练方法 DOBF，它利用编程语言的结构特性，对模型进行预训练，以恢复混淆的源代码的原始版本，表明使用 DOBF 预训练的模型在多种下游任务上具有明显的性能优势，例如在无监督代码翻译和自然语言代码搜索方面分别提供了多达 13% 和 24% 的相对改进。

DOBF: 面向编程语言的反混淆预训练目标