The latest paradigm shift in software development brings in the innovation
and automation afforded by Large Language Models (LLMs), showcased by
Generative Pre-trained Transformer (GPT), which has shown remarkable capacity
to generate code autonomously, significantly reducing the manual effort
required for various programming tasks. Although, the potential benefits of
LLM-generated code are vast, most notably in efficiency and rapid prototyping,
as LLMs become increasingly integrated into the software development lifecycle
and hence the supply chain, complex and multifaceted challenges arise as the
code generated from these language models carry profound questions on quality
and correctness. Research is required to comprehensively explore these critical
concerns surrounding LLM-generated code.
In this paper, we propose a novel solution called metamorphic prompt testing
to address these challenges. Our intuitive observation is that intrinsic
consistency always exists among correct code pieces but may not exist among
flawed code pieces, so we can detect flaws in the code by detecting
inconsistencies. Therefore, we can vary a given prompt to multiple prompts with
paraphrasing, and to ask the LLM to acquire multiple versions of generated
code, so that we can validate whether the semantic relations still hold in the
acquired code through cross-validation. Our evaluation on HumanEval shows that
metamorphic prompt testing is able to detect 75 percent of the erroneous
programs generated by GPT-4, with a false positive rate of 8.6 percent.

我们提出了一种称为变质提示测试的新颖解决方案，用于解决由大型语言模型生成的代码质量和正确性所引发的挑战，并在 HumanEval 评估中显示，该方法能够检测到由 GPT-4 生成的错误程序的 75％，误报率为 8.6％。