The Software Naturalness hypothesis argues that programming languages can be
understood through the same techniques used in natural language processing. We
explore this hypothesis through the use of a pre-trained transformer-based
language model to perform code analysis tasks. Present approaches to code
analysis depend heavily on features derived from the Abstract Syntax Tree (AST)
while our transformer-based language models work on raw source code. This work
is the first to investigate whether such language models can discover AST
features automatically. To achieve this, we introduce a sequence labeling task
that directly probes the language models understanding of AST. Our results show
that transformer based language models achieve high accuracy in the AST tagging
task. Furthermore, we evaluate our model on a software vulnerability
identification task. Importantly, we show that our approach obtains
vulnerability identification results comparable to graph based approaches that
rely heavily on compilers for feature extraction.

本文探讨了利用预训练的基于 transformer 的语言模型来对代码进行分析任务的能力，通过对 transformer-based language model 进行 AST 标记任务，得出结论表明其在此任务上的表现较好，并在软件漏洞识别任务中与依赖于编译器的基于图形的方法相比，取得了可比拟的成果。