Programming-based Pre-trained Language Models (PPLMs) such as CodeBERT have
achieved great success in many downstream code-related tasks. Since the memory
and computational complexity of self-attention in the Transformer grow
quadratically with the sequence length, PPLMs typically limit the code length
to 512. However, codes in real-world applications are generally long, such as
code searches, which cannot be processed efficiently by existing PPLMs. To
solve this problem, in this paper, we present SASA, a Structure-Aware Sparse
Attention mechanism, which reduces the complexity and improves performance for
long code understanding tasks. The key components in SASA are top-$k$ sparse
attention and Abstract Syntax Tree (AST)-based structure-aware attention. With
top-$k$ sparse attention, the most crucial attention relation can be obtained
with a lower computational cost. As the code structure represents the logic of
the code statements, which is a complement to the code sequence
characteristics, we further introduce AST structures into attention. Extensive
experiments on CodeXGLUE tasks show that SASA achieves better performance than
the competing baselines.

本文提出了一种名为 SASA 的结构感知稀疏注意机制用于长代码理解任务。SASA 采用 top-k 稀疏关注机制和基于抽象语法树的结构感知关注机制，能够高效降低计算成本并处理长代码，实现了优于竞争基准模型的表现。