In this paper, we propose a fully automated method, named ACCA, to evaluate
the correctness of AI-generated code for security purposes. The method uses
symbolic execution to assess whether the AI-generated code behaves as a
reference implementation. We use ACCA to assess four state-of-the-art models
trained to generate security-oriented assembly code and compare the results of
the evaluation with different baseline solutions, including output similarity
metrics, widely used in the field, and the well-known ChatGPT, the AI-powered
language model developed by OpenAI. Our experiments show that our method
outperforms the baseline solutions and assesses the correctness of the
AI-generated code similar to the human-based evaluation, which is considered
the ground truth for the assessment in the field. Moreover, ACCA has a very
strong correlation with human evaluation (Pearson's correlation coefficient
r=0.84 on average). Finally, since it is a fully automated solution that does
not require any human intervention, the proposed method performs the assessment
of every code snippet in ~0.17s on average, which is definitely lower than the
average time required by human analysts to manually inspect the code, based on
our experience.

本文提出了一种名为 ACCA 的完全自动化方法，用于评估用于安全目的的 AI 生成的代码的正确性。该方法使用符号执行来评估 AI 生成的代码是否与参考实现相同，并通过比较结果与领域中广泛使用的不同基准解决方案以及 OpenAI 开发的人工智能语言模型 ChatGPT，来评估四种用于生成面向安全的汇编代码的现有模型。我们的实验表明，我们的方法优于基准解决方案，并且与人工评估类似地评估 AI 生成的代码的正确性，而人工评估在该领域被认为是基准事实。此外，ACCA 与人工评估具有非常强的相关性（平均值皮尔逊相关系数 r=0.84）。最后，由于该方法是完全自动化的，不需要任何人工干预，所以所提出的方法对于评估每个代码片段的时间平均为～0.17 秒，这肯定低于人工分析员根据我们的经验手动检查代码所需的平均时间。