Pre-trained models of source code have recently been successfully applied to
a wide variety of Software Engineering tasks; they have also seen some
practical adoption in practice, e.g. for code completion. Yet, we still know
very little about what these pre-trained models learn about source code. In
this article, we use probing--simple diagnostic tasks that do not further train
the models--to discover to what extent pre-trained models learn about specific
aspects of source code. We use an extensible framework to define 15 probing
tasks that exercise surface, syntactic, structural and semantic characteristics
of source code. We probe 8 pre-trained source code models, as well as a natural
language model (BERT) as our baseline. We find that models that incorporate
some structural information (such as GraphCodeBERT) have a better
representation of source code characteristics. Surprisingly, we find that for
some probing tasks, BERT is competitive with the source code models, indicating
that there are ample opportunities to improve source-code specific pre-training
on the respective code characteristics. We encourage other researchers to
evaluate their models with our probing task suite, so that they may peer into
the hidden layers of the models and identify what intrinsic code
characteristics are encoded.

我们使用探测任务来检验预训练模型对源代码的学习程度，并发现结构信息对源代码的特征表示具有改进效果，同时发现 BERT 在某些探测任务上与源代码模型竞争力相当，暗示改进源代码特定的预训练有丰富的机会。