In the field of source code processing, the transformer-based representation
models have shown great powerfulness and have achieved state-of-the-art (SOTA)
performance in many tasks. Although the transformer models process the
sequential source code, pieces of evidence show that they may capture the
structural information (\eg, in the syntax tree, data flow, control flow, \etc)
as well. We propose the aggregated attention score, a method to investigate the
structural information learned by the transformer. We also put forward the
aggregated attention graph, a new way to extract program graphs from the
pre-trained models automatically. We measure our methods from multiple
perspectives. Furthermore, based on our empirical findings, we use the
automatically extracted graphs to replace those ingenious manual designed
graphs in the Variable Misuse task. Experimental results show that the semantic
graphs we extracted automatically are greatly meaningful and effective, which
provide a new perspective for us to understand and use the information
contained in the model.

文章提出了利用聚合注意力得分和注意力图从预训练模型自动提取程序图的方法，并将该方法应用于变量误用任务中，证明了自动提取的程序图具有极高的有意义和有效性。

Transformer 模型对于源代码的学习

What does Transformer learn about source code?

Human face recognition is one of the most important research areas in
biometrics. However, the robust face recognition under a drastic change of the
facial pose, expression, and illumination is a big challenging problem for its
practical application. Such variations make face recognition more difficult. In
this paper, we propose a novel face recognition method, called Attentional
Feature-pair Relation Network (AFRN), which represents the face by the relevant
pairs of local appearance block features with their attention scores. The AFRN
represents the face by all possible pairs of the 9x9 local appearance block
features, the importance of each pair is considered by the attention map that
is obtained from the low-rank bilinear pooling, and each pair is weighted by
its corresponding attention score. To increase the accuracy, we select top-K
pairs of local appearance block features as relevant facial information and
drop the remaining irrelevant. The weighted top-K pairs are propagated to
extract the joint feature-pair relation by using bilinear attention network. In
experiments, we show the effectiveness of the proposed AFRN and achieve the
outstanding performance in the 1:1 face verification and 1:N face
identification tasks compared to existing state-of-the-art methods on the
challenging LFW, YTF, CALFW, CPLFW, CFP, AgeDB, IJB-A, IJB-B, and IJB-C
datasets.

本论文提出了一种名为 AFRN 的人脸识别方法，它是通过考虑注意力分数来表示本地外观块特征的相关配对，以增加准确性并通过使用双线性注意力网络提取联合特征 - 对关系。在实验中，该方法在诸多数据集上的表现均超越了现有的最先进的方法。