In recent years, generative AI has undergone major advancements,
demonstrating significant promise in augmenting human productivity. Notably,
large language models (LLM), with ChatGPT-4 as an example, have drawn
considerable attention. Numerous articles have examined the impact of LLM-based
tools on human productivity in lab settings and designed tasks or in
observational studies. Despite recent advances, field experiments applying
LLM-based tools in realistic settings are limited. This paper presents the
findings of a field randomized controlled trial assessing the effectiveness of
LLM-based tools in providing unmonitored support services for information
retrieval.

通过一个领域性的随机控制试验，评估基于大型语言模型的工具在提供无监控支持服务方面的有效性。

聊天机器人的人工智能革命：来自一项随机对照实验的证明

AI Revolution on Chat Bot: Evidence from a Randomized Controlled  Experiment

Large language models (LLMs) have been touted to enable increased
productivity in many areas of today's work life. Scientific research as an area
of work is no exception: the potential of LLM-based tools to assist in the
daily work of scientists has become a highly discussed topic across
disciplines. However, we are only at the very onset of this subject of study.
It is still unclear how the potential of LLMs will materialise in research
practice. With this study, we give first empirical evidence on the use of LLMs
in the research process. We have investigated a set of use cases for LLM-based
tools in scientific research, and conducted a first study to assess to which
degree current tools are helpful. In this paper we report specifically on use
cases related to software engineering, such as generating application code and
developing scripts for data analytics. While we studied seemingly simple use
cases, results across tools differ significantly. Our results highlight the
promise of LLM-based tools in general, yet we also observe various issues,
particularly regarding the integrity of the output these tools provide.

该研究报告关注大型语言模型在科学研究中的应用，调查了使用 LLM 工具的若干实际案例，特别是与软件工程相关的用例，结果显示了大型语言模型工具的潜力和问题。