Empirical software engineering research on production systems has brought
forth a better understanding of the software engineering process for
practitioners and researchers alike. However, only a small subset of production
systems is studied, limiting the impact of this research. While software
engineering practitioners benefit from replicating research on their own data,
this poses its own set of challenges, since performing replications requires a
deep understanding of research methodologies and subtle nuances in software
engineering data. Given that large language models (LLMs), such as GPT-4, show
promise in tackling both software engineering- and science-related tasks, these
models could help democratize empirical software engineering research.
In this paper, we examine LLMs' abilities to perform replications of
empirical software engineering research on new data. We specifically study
their ability to surface assumptions made in empirical software engineering
research methodologies, as well as their ability to plan and generate code for
analysis pipelines on seven empirical software engineering papers. We perform a
user study with 14 participants with software engineering research expertise,
who evaluate GPT-4-generated assumptions and analysis plans (i.e., a list of
module specifications) from the papers. We find that GPT-4 is able to surface
correct assumptions, but struggle to generate ones that reflect common
knowledge about software engineering data. In a manual analysis of the
generated code, we find that the GPT-4-generated code contains the correct
high-level logic, given a subset of the methodology. However, the code contains
many small implementation-level errors, reflecting a lack of software
engineering knowledge. Our findings have implications for leveraging LLMs for
software engineering research as well as practitioner data scientists in
software teams.

本研究旨在考察大型语言模型（如 GPT-4）在新数据上执行经验软件工程研究的复制能力，发现 GPT-4 能够提供正确的假设，但难以生成反映软件工程数据的常规知识的假设，同时在生成的代码中存在实现层面错误，显示出对软件工程知识的欠缺。这些发现对于利用 LLMs 进行软件工程研究以及软件团队中的数据科学家具有重要意义。