Recent results of machine learning for automatic vulnerability detection have
been very promising indeed: Given only the source code of a function $f$,
models trained by machine learning techniques can decide if $f$ contains a
security flaw with up to 70% accuracy.
But how do we know that these results are general and not specific to the
datasets? To study this question, researchers proposed to amplify the testing
set by injecting semantic preserving changes and found that the model's
accuracy significantly drops. In other words, the model uses some unrelated
features during classification. In order to increase the robustness of the
model, researchers proposed to train on amplified training data, and indeed
model accuracy increased to previous levels.
In this paper, we replicate and continue this investigation, and provide an
actionable model benchmarking methodology to help researchers better evaluate
advances in machine learning for vulnerability detection. Specifically, we
propose (i) a cross validation algorithm, where a semantic preserving
transformation is applied during the amplification of either the training set
or the testing set, and (ii) the amplification of the testing set with code
snippets where the vulnerabilities are fixed. Using 11 transformations, 3 ML
techniques, and 2 datasets, we find that the improved robustness only applies
to the specific transformations used during training data amplification. In
other words, the robustified models still rely on unrelated features for
predicting the vulnerabilities in the testing data. Additionally, we find that
the trained models are unable to generalize to the modified setting which
requires to distinguish vulnerable functions from their patches.

本文研究了机器学习用于自动漏洞检测的泛化问题，并提出了一种可行的模型基准测试方法来帮助研究人员更好地评估漏洞检测机器学习的进展。研究表明，当前模型对于特定的训练数据扩增方式表现出了较好的鲁棒性。