In the field of source code processing, the transformer-based representation
models have shown great powerfulness and have achieved state-of-the-art (SOTA)
performance in many tasks. Although the transformer models process the
sequential source code, pieces of evidence show that they may capture the
structural information (\eg, in the syntax tree, data flow, control flow, \etc)
as well. We propose the aggregated attention score, a method to investigate the
structural information learned by the transformer. We also put forward the
aggregated attention graph, a new way to extract program graphs from the
pre-trained models automatically. We measure our methods from multiple
perspectives. Furthermore, based on our empirical findings, we use the
automatically extracted graphs to replace those ingenious manual designed
graphs in the Variable Misuse task. Experimental results show that the semantic
graphs we extracted automatically are greatly meaningful and effective, which
provide a new perspective for us to understand and use the information
contained in the model.

文章提出了利用聚合注意力得分和注意力图从预训练模型自动提取程序图的方法，并将该方法应用于变量误用任务中，证明了自动提取的程序图具有极高的有意义和有效性。

Transformer 模型对于源代码的学习

What does Transformer learn about source code?

There is an emerging interest in the application of natural language
processing models to source code processing tasks. One of the major problems in
applying deep learning to software engineering is that source code often
contains a lot of rare identifiers, resulting in huge vocabularies. We propose
a simple, yet effective method, based on identifier anonymization, to handle
out-of-vocabulary (OOV) identifiers. Our method can be treated as a
preprocessing step and, therefore, allows for easy implementation. We show that
the proposed OOV anonymization method significantly improves the performance of
the Transformer in two code processing tasks: code completion and bug fixing.

本文介绍了一种使用标识符匿名化解决深度学习在软件工程中处理源代码中罕见标识符的方法，并表明这种方法显著提高了 Transformer 在代码完成和错误修复等两个任务中的性能。

深度学习中处理源代码未识别标识符的简单方法

A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep  Learning for Source Code

Source code processing heavily relies on the methods widely used in natural
language processing (NLP), but involves specifics that need to be taken into
account to achieve higher quality. An example of this specificity is that the
semantics of a variable is defined not only by its name but also by the
contexts in which the variable occurs. In this work, we develop dynamic
embeddings, a recurrent mechanism that adjusts the learned semantics of the
variable when it obtains more information about the variable's role in the
program. We show that using the proposed dynamic embeddings significantly
improves the performance of the recurrent neural network, in code completion
and bug fixing tasks.

本研究提出了动态嵌入，这是一种递归机制，能够依据变量在程序中的角色来调整学习到的语义，该方法在代码自动补全和错误修复任务中，显著提高了循环神经网络的性能。

关于循环神经网络变量嵌入在源代码中的研究

On the Embeddings of Variables in Recurrent Neural Networks for Source  Code

Initially developed for natural language processing (NLP), Transformers are
now widely used for source code processing, due to the format similarity
between source code and text. In contrast to natural language, source code is
strictly structured, i.e., it follows the syntax of the programming language.
Several recent works develop Transformer modifications for capturing syntactic
information in source code. The drawback of these works is that they do not
compare to each other and consider different tasks. In this work, we conduct a
thorough empirical study of the capabilities of Transformers to utilize
syntactic information in different tasks. We consider three tasks (code
completion, function naming and bug fixing) and re-implement different
syntax-capturing modifications in a unified framework. We show that
Transformers are able to make meaningful predictions based purely on syntactic
information and underline the best practices of taking the syntactic
information into account for improving the performance of the model.

本文研究了如何使用 Transformers 提取和利用源代码中的句法信息，完成代码自动补全、函数命名和 bug 修复等任务，实现了更好的模型性能。