Bugs are essential in software engineering; many research studies in the past
decades have been proposed to detect, localize, and repair bugs in software
systems. Effectiveness evaluation of such techniques requires complex bugs,
i.e., those that are hard to detect through testing and hard to repair through
debugging. From the classic software engineering point of view, a
hard-to-repair bug differs from the correct code in multiple locations, making
it hard to localize and repair. Hard-to-detect bugs, on the other hand,
manifest themselves under specific test inputs and reachability conditions.
These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs,
are mostly aligned; a bug generation technique can change multiple statements
to be covered only under a specific set of inputs. However, these two
objectives are conflicting for learning-based techniques: A bug should have a
similar code representation to the correct code in the training data to
challenge a bug prediction model to distinguish them. The hard-to-repair bug
definition remains the same but with a caveat: the more a bug differs from the
original code (at multiple locations), the more distant their representations
are and easier to be detected. We propose BugFarm, to transform arbitrary code
into multiple complex bugs. BugFarm leverages LLMs to mutate code in multiple
locations (hard-to-repair). To ensure that multiple modifications do not
notably change the code representation, BugFarm analyzes the attention of the
underlying model and instructs LLMs to only change the least attended locations
(hard-to-detect). Our comprehensive evaluation of 320k+ bugs from over 2.5M
mutants generated by BugFarm and two alternative approaches demonstrates our
superiority in generating bugs that are hard to detect by learning-based bug
prediction approaches and hard to repair by SOTA learning-based program repair
technique.

草案提出了 BugFarm，用于将任意代码转化为多个复杂的错误。BugFarm 利用 LLM（语言模型）在多个位置（难以修复）突变代码。为了确保多个修改不会显着改变代码的表示，BugFarm 分析了底层模型的注意力，并指示 LLMs 仅更改注意力最低的位置（难以检测）。我们对 BugFarm 生成的 320k + 错误进行了全面评估，这些错误是通过学习型错误预测方法难以检测到的，并且通过 SOTA（优秀的）学习型程序修复技术难以修复的。

大语言模型时代的自动漏洞生成

Automated Bug Generation in the era of Large Language Models

This paper targets the problem of speech act detection in conversations about
bug repair. We conduct a "Wizard of Oz" experiment with 30 professional
programmers, in which the programmers fix bugs for two hours, and use a
simulated virtual assistant for help. Then, we use an open coding manual
annotation procedure to identify the speech act types in the conversations.
Finally, we train and evaluate a supervised learning algorithm to automatically
detect the speech act types in the conversations. In 30 two-hour conversations,
we made 2459 annotations and uncovered 26 speech act types. Our automated
detection achieved 69% precision and 50% recall. The key application of this
work is to advance the state of the art for virtual assistants in software
engineering. Virtual assistant technology is growing rapidly, though
applications in software engineering are behind those in other areas, largely
due to a lack of relevant data and experiments. This paper targets this problem
in the area of developer Q/A conversations about bug repair.

基于 “巫师奥兹” 的实验，利用自然语言处理标注工具自动检测软件工程中开发者关于漏洞修复方面的交流中的语音行为类型，可用于提高虚拟助手在软件工程中的自动交流能力。