Pre-trained code models lead the era of code intelligence. Many models have
been designed with impressive performance recently. However, one important
problem, data augmentation for code data that automatically helps developers
prepare training data lacks study in the field of code learning. In this paper,
we introduce a general data augmentation framework, GenCode, to enhance the
training of code understanding models. GenCode follows a
generation-and-selection paradigm to prepare useful training codes.
Specifically, it uses code transformation techniques to generate new code
candidates first and then selects important ones as the training data by
importance metrics. To evaluate the effectiveness of GenCode with a general
importance metric -- loss value, we conduct experiments on four code
understanding tasks (e.g., code clone detection) and three pre-trained code
models (e.g., CodeT5). Compared to the state-of-the-art (SOTA) code
augmentation method, MixCode, GenCode produces code models with 2.92% higher
accuracy and 4.90% robustness on average.

通过引入通用的数据增强框架 GenCode，该论文提出了一种用于增强代码理解模型训练的方法，通过代码转换技术生成新的代码候选项，并利用重要性度量选择重要的代码作为训练数据，实验证明 GenCode 相比 MixCode 增强方法，在平均准确率上提高了 2.92% 并且在鲁棒性方面提高了 4.90%。

基于神经网络的代码理解的重要性导向数据增强

Importance Guided Data Augmentation for Neural-Based Code Understanding

Recent breakthroughs in pre-trained code models, such as CodeBERT and Codex,
have shown their superior performance in various downstream tasks. The
correctness and unambiguity of API usage among these code models are crucial
for achieving desirable program functionalities, requiring them to learn
various API fully qualified names structurally and semantically. Recent studies
reveal that even state-of-the-art pre-trained code models struggle with
suggesting the correct APIs during code generation. However, the reasons for
such poor API usage performance are barely investigated. To address this
challenge, we propose using knowledge probing as a means of interpreting code
models, which uses cloze-style tests to measure the knowledge stored in models.
Our comprehensive study examines a code model's capability of understanding API
fully qualified names from two different perspectives: API call and API import.
Specifically, we reveal that current code models struggle with understanding
API names, with pre-training strategies significantly affecting the quality of
API name learning. We demonstrate that natural language context can assist code
models in locating Python API names and generalize Python API name knowledge to
unseen data. Our findings provide insights into the limitations and
capabilities of current pre-trained code models, and suggest that incorporating
API structure into the pre-training process can improve automated API usage and
code representations. This work provides significance for advancing code
intelligence practices and direction for future studies. All experiment
results, data and source code used in this work are available at
https://doi.org/10.5281/zenodo.7902072.

最近的研究突破，如 CodeBERT 和 Codex 等，已经显示出其在各种下游任务中的优越性能。这些代码模型在实现期望的程序功能方面的 API 使用的正确性和明确性对于他们至关重要，要求它们在结构上和语义上学习各种 API 的全限定名称。最近的研究表明，即使是最先进的预训练代码模型在代码生成过程中也难以提供正确的 API 建议。然而，对于这种糟糕的 API 使用性能的原因几乎没有进行研究。为了应对这一挑战，我们提出使用知识探测作为解释代码模型的手段，该方法使用填空式测试来衡量模型中存储的知识。我们综合研究了代码模型从两个不同角度理解 API 全限定名称的能力：API 调用和 API 导入。具体而言，我们发现当前的代码模型在理解 API 名称方面面临困难，而预训练策略对 API 名称学习的质量产生显著影响。我们证明了自然语言上下文可以帮助代码模型定位 Python API 名称，并将 Python API 名称知识推广到未见数据中。我们的发现提供了有关当前预训练代码模型的局限性和能力的见解，并暗示在预训练过程中结构 API 可以改进自动化 API 使用和代码表示。这项工作对于推进代码智能实践具有重要意义，为未来的研究提供了方向。

弹窗测验！预训练代码模型是否具备正确的 API 命名知识？

Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API  Names?

Recently, fine-tuning pre-trained code models such as CodeBERT on downstream
tasks has achieved great success in many software testing and analysis tasks.
While effective and prevalent, fine-tuning the pre-trained parameters incurs a
large computational cost. In this paper, we conduct an extensive experimental
study to explore what happens to layer-wise pre-trained representations and
their encoded code knowledge during fine-tuning. We then propose efficient
alternatives to fine-tune the large pre-trained code model based on the above
findings. Our experimental study shows that (1) lexical, syntactic and
structural properties of source code are encoded in the lower, intermediate,
and higher layers, respectively, while the semantic property spans across the
entire model. (2) The process of fine-tuning preserves most of the code
properties. Specifically, the basic code properties captured by lower and
intermediate layers are still preserved during fine-tuning. Furthermore, we
find that only the representations of the top two layers change most during
fine-tuning for various downstream tasks. (3) Based on the above findings, we
propose Telly to efficiently fine-tune pre-trained code models via layer
freezing. The extensive experimental results on five various downstream tasks
demonstrate that training parameters and the corresponding time cost are
greatly reduced, while performances are similar or better. Replication package
including source code, datasets, and online Appendix is available at:
https://github.com/DeepSoftwareAnalytics/Telly.

该研究通过分层预训练表示的实验研究发现了不同层数与源代码信息的关系，从而提出了 Telly 技术，通过层冻结来有效地微调预训练代码模型，从而降低了学习参数和时间成本。