Code pre-trained models (CodePTMs) have recently demonstrated significant
success in code intelligence. To interpret these models, some probing methods
have been applied. However, these methods fail to consider the inherent
characteristics of codes. In this paper, to address the problem, we propose a
novel probing method CAT-probing to quantitatively interpret how CodePTMs
attend code structure. We first denoise the input code sequences based on the
token types pre-defined by the compilers to filter those tokens whose attention
scores are too small. After that, we define a new metric CAT-score to measure
the commonality between the token-level attention scores generated in CodePTMs
and the pair-wise distances between corresponding AST nodes. The higher the
CAT-score, the stronger the ability of CodePTMs to capture code structure. We
conduct extensive experiments to integrate CAT-probing with representative
CodePTMs for different programming languages. Experimental results show the
effectiveness of CAT-probing in CodePTM interpretation. Our codes and data are
publicly available at this https URL.

本研究通过提出一种新的 CAT-probing 探针方法，定量解释 CodePTMs 如何关注代码结构，通过过滤编译器预定义的令牌类型的令牌，定义一个新的度量 CAT-score 来衡量令牌级别注意力得分与相应 AST 节点之间的成对距离之间的公共度量，本文证明了 CAT-probing 的有效性。