Detecting vulnerabilities within compiled binaries is challenging due to lost
high-level code structures and other factors such as architectural
dependencies, compilers, and optimization options. To address these obstacles,
this research explores vulnerability detection by using natural language
processing (NLP) embedding techniques with word2vec, BERT, and RoBERTa to learn
semantics from intermediate representation (LLVM) code. Long short-term memory
(LSTM) neural networks were trained on embeddings from encoders created using
approximately 118k LLVM functions from the Juliet dataset. This study is
pioneering in its comparison of word2vec models with multiple bidirectional
transformer (BERT, RoBERTa) embeddings built using LLVM code to train neural
networks to detect vulnerabilities in compiled binaries. word2vec Continuous
Bag of Words (CBOW) models achieved 92.3% validation accuracy in detecting
vulnerabilities, outperforming word2vec Skip-Gram, BERT, and RoBERTa. This
suggests that complex contextual NLP embeddings may not provide advantages over
simpler word2vec models for this task when a limited number (e.g. 118K) of data
samples are used to train the bidirectional transformer-based models. The
comparative results provide novel insights into selecting optimal embeddings
for learning compiler-independent semantic code representations to advance
machine learning detection of vulnerabilities in compiled binaries.

利用自然语言处理（NLP）中的 word2vec、BERT 和 RoBERTa 等嵌入技术，通过学习中间表示（LLVM）代码的语义，利用长短期记忆（LSTM）神经网络对 Juliet 数据集中近 118k 个 LLVM 函数的嵌入进行训练，进而检测编译二进制文件中的漏洞。研究结果显示，与复杂的上下文 NLP 嵌入相比，word2vec 连续词袋（CBOW）模型在检测漏洞方面取得了 92.3% 的验证准确率，表明当用来训练双向变压器模型的数据样本数量有限（例如 118K）时，简单的 word2vec 模型在这个任务中可能更有优势。这些比较结果为选择用于学习编译器独立语义代码表示的最佳嵌入提供了新的见解，推动了编译二进制文件中漏洞的机器学习检测的发展。