There is an increasing trend to mine vulnerabilities from software
repositories and use machine learning techniques to automatically detect
software vulnerabilities. A fundamental but unresolved research question is:
how do different factors in the mining and learning process impact the accuracy
of identifying vulnerabilities in software projects of varying characteristics?
Substantial research has been dedicated in this area, including source code
static analysis, software repository mining, and NLP-based machine learning.
However, practitioners lack experience regarding the key factors for building a
baseline model of the state-of-the-art. In addition, there lacks of experience
regarding the transferability of the vulnerability signatures from project to
project. This study investigates how the combination of different vulnerability
features and three representative machine learning models impact the accuracy
of vulnerability detection in 17 real-world projects. We examine two types of
vulnerability representations: 1) code features extracted through NLP with
varying tokenization strategies and three different embedding techniques
(bag-of-words, word2vec, and fastText) and 2) a set of eight architectural
metrics that capture the abstract design of the software systems. The three
machine learning algorithms include a random forest model, a support vector
machines model, and a residual neural network model. The analysis shows a
recommended baseline model with signatures extracted through bag-of-words
embedding, combined with the random forest, consistently increases the
detection accuracy by about 4% compared to other combinations in all 17
projects. Furthermore, we observe the limitation of transferring vulnerability
signatures across domains based on our experiments.

研究了软件项目中不同因素对于识别漏洞的准确性的影响，通过挖掘软件库中的漏洞并使用机器学习技术进行自动检测。通过实验发现，将基于词袋模型的签名与随机森林模型相结合，在 17 个真实世界项目中能提高 4% 的检测准确率，并观察到在跨域中转移漏洞签名的局限性。