In the field of source code processing, the transformer-based representation
models have shown great powerfulness and have achieved state-of-the-art (SOTA)
performance in many tasks. Although the transformer models process the
sequential source code, pieces of evidence show that they may capture the
structural information (\eg, in the syntax tree, data flow, control flow, \etc)
as well. We propose the aggregated attention score, a method to investigate the
structural information learned by the transformer. We also put forward the
aggregated attention graph, a new way to extract program graphs from the
pre-trained models automatically. We measure our methods from multiple
perspectives. Furthermore, based on our empirical findings, we use the
automatically extracted graphs to replace those ingenious manual designed
graphs in the Variable Misuse task. Experimental results show that the semantic
graphs we extracted automatically are greatly meaningful and effective, which
provide a new perspective for us to understand and use the information
contained in the model.

文章提出了利用聚合注意力得分和注意力图从预训练模型自动提取程序图的方法，并将该方法应用于变量误用任务中，证明了自动提取的程序图具有极高的有意义和有效性。