Large language models (LLMs) show impressive capabilities, matching and
sometimes exceeding human performance in many domains. This study explores the
potential of LLMs to augment judgement in forecasting tasks. We evaluated the
impact on forecasting accuracy of two GPT-4-Turbo assistants: one designed to
provide high-quality advice ('superforecasting'), and the other designed to be
overconfident and base-rate-neglecting. Participants (N = 991) had the option
to consult their assigned LLM assistant throughout the study, in contrast to a
control group that used a less advanced model (DaVinci-003) without direct
forecasting support. Our preregistered analyses reveal that LLM augmentation
significantly enhances forecasting accuracy by 23% across both types of
assistants, compared to the control group. This improvement occurs despite the
superforecasting assistant's higher accuracy in predictions, indicating the
augmentation's benefit is not solely due to model prediction accuracy.
Exploratory analyses showed a pronounced effect in one forecasting item,
without which we find that the superforecasting assistant increased accuracy by
43%, compared with 28% for the biased assistant. We further examine whether LLM
augmentation disproportionately benefits less skilled forecasters, degrades the
wisdom-of-the-crowd by reducing prediction diversity, or varies in
effectiveness with question difficulty. Our findings do not consistently
support these hypotheses. Our results suggest that access to an LLM assistant,
even a biased one, can be a helpful decision aid in cognitively demanding tasks
where the answer is not known at the time of interaction.

该研究探讨了大型语言模型（LLMs）在预测任务中增强判断能力的潜力，研究结果表明，与对照组相比，LLM 辅助可以显著提高预测准确性，即使是面对回答不确定的认知要求高的任务，使用 LLM 助手（即使是倾向性的助手）仍然可以作为有益的决策辅助。

AI 增强预测：LLM 助理提高人类预测准确性

AI-Augmented Predictions: LLM Assistants Improve Human Forecasting  Accuracy

This scoping survey focuses on our current understanding of the design space
for task-oriented LLM systems and elaborates on definitions and relationships
among the available design parameters. The paper begins by defining a minimal
task-oriented LLM system and exploring the design space of such systems through
a thought experiment contemplating the performance of diverse LLM system
configurations (involving single LLMs, single LLM-based agents, and multiple
LLM-based agent systems) on a complex software development task and
hypothesizes the results. We discuss a pattern in our results and formulate
them into three conjectures. While these conjectures may be partly based on
faulty assumptions, they provide a starting point for future research. The
paper then surveys a select few design parameters: covering and organizing
research in LLM augmentation, prompting techniques, and uncertainty estimation,
and discussing their significance. The paper notes the lack of focus on
computational and energy efficiency in evaluating research in these areas. Our
survey findings provide a basis for developing the concept of linear and
non-linear contexts, which we define and use to enable an agent-centric
projection of prompting techniques providing a lens through which prompting
techniques can be viewed as multi-agent systems. The paper discusses the
implications of this lens, for the cross-pollination of research between LLM
prompting and LLM-based multi-agent systems; and also, for the generation of
synthetic training data based on existing prompting techniques in research. In
all, the scoping survey presents seven conjectures that can help guide future
research efforts.

这项调研重点关注任务导向的 LLM 系统的设计空间，通过思考实验对不同的 LLM 系统配置性能进行探索，并对 LLM 增强、提示技术和不确定性估计等设计参数进行讨论，提出了七个猜想以指导未来的研究努力。