Representational Similarity Analysis is a method from cognitive neuroscience,
which helps in comparing representations from two different sources of data. In
this paper, we propose using Representational Similarity Analysis to probe the
semantic grounding in language models of code. We probe representations from
the CodeBERT model for semantic grounding by using the data from the IBM
CodeNet dataset. Through our experiments, we show that current pre-training
methods do not induce semantic grounding in language models of code, and
instead focus on optimizing form-based patterns. We also show that even a
little amount of fine-tuning on semantically relevant tasks increases the
semantic grounding in CodeBERT significantly. Our ablations with the input
modality to the CodeBERT model show that using bimodal inputs (code and natural
language) over unimodal inputs (only code) gives better semantic grounding and
sample efficiency during semantic fine-tuning. Finally, our experiments with
semantic perturbations in code reveal that CodeBERT is able to robustly
distinguish between semantically correct and incorrect code.

本文利用相似性比较方法研究语言代码模型的语义基础，并通过 IBM CodeNet 数据集研究 CodeBERT 的语义基础。实验表明，现有的预训练方法并未引入语言代码模型的语义基础，而是集中优化基于形式的模式，而进行少量的语义相关任务的微调，可显著提高 CodeBERT 的语义基础。使用双模态输入比单模态输入更好地提高了 CodeBERT 的语义基础，并具有更高样本效率。最后，实验表明，CodeBERT 能够稳健地区分语义上正确和不正确的代码。