Pretrained models of code, such as CodeBERT and CodeT5, have become popular
choices for code understanding and generation tasks. Such models tend to be
large and require commensurate volumes of training data, which are rarely
available for downstream tasks. Instead, it has become popular to train models
with far larger but less realistic datasets, such as functions with
artificially injected bugs. Models trained on such data, however, tend to only
perform well on similar data, while underperforming on real world programs. In
this paper, we conjecture that this discrepancy stems from the presence of
distracting samples that steer the model away from the real-world task
distribution. To investigate this conjecture, we propose an approach for
identifying the subsets of these large yet unrealistic datasets that are most
similar to examples in real-world datasets based on their learned
representations. Our approach extracts high-dimensional embeddings of both
real-world and artificial programs using a neural model and scores artificial
samples based on their distance to the nearest real-world sample. We show that
training on only the nearest, representationally most similar samples while
discarding samples that are not at all similar in representations yields
consistent improvements across two popular pretrained models of code on two
code understanding tasks. Our results are promising, in that they show that
training models on a representative subset of an unrealistic dataset can help
us harness the power of large-scale synthetic data generation while preserving
downstream task performance. Finally, we highlight the limitations of applying
AI models for predicting vulnerabilities and bugs in real-world applications

通过提取高维嵌入并评分人工样本相对于最接近的真实世界样本的距离，我们展示了在基于代码的两个流行预训练模型上训练仅包含表示最相似样本且舍弃与表示完全不相似样本的方法对两个代码理解任务取得了一致的改进。

从非现实数据学习缺陷预测

Learning Defect Prediction from Unrealistic Data

Program optimization is the process of modifying software to execute more
efficiently. Superoptimizers attempt to find the optimal program by employing
significantly more expensive search and constraint solving techniques.
Generally, these methods do not scale well to programs in real development
scenarios, and as a result, superoptimization has largely been confined to
small-scale, domain-specific, and/or synthetic program benchmarks. In this
paper, we propose a framework to learn to superoptimize real-world programs by
using neural sequence-to-sequence models. We created a dataset consisting of
over 25K real-world x86-64 assembly functions mined from open-source projects
and propose an approach, Self Imitation Learning for Optimization (SILO) that
is easy to implement and outperforms a standard policy gradient learning
approach on our dataset. Our method, SILO, superoptimizes 5.9% of our test set
when compared with the gcc version 10.3 compiler's aggressive optimization
level -O3. We also report that SILO's rate of superoptimization on our test set
is over five times that of a standard policy gradient approach and a model
pre-trained on compiler optimization demonstration.

本文提出了一种使用神经序列到序列模型来学习超级优化现实世界程序的框架，并介绍了一种易于实现的方法（SILO）来增强优化，该方法在测试集上优化了 5.9％，超过了标准策略梯度方法和模型预训练的优化演示。