Robust aggregation integrates predictions from multiple experts without
knowledge of the experts' information structures. Prior work assumes experts
are Bayesian, providing predictions as perfect posteriors based on their
signals. However, real-world experts often deviate systematically from Bayesian
reasoning. Our work considers experts who tend to ignore the base rate. We find
that a certain degree of base rate neglect helps with robust forecast
aggregation.
Specifically, we consider a forecast aggregation problem with two experts who
each predict a binary world state after observing private signals. Unlike
previous work, we model experts exhibiting base rate neglect, where they
incorporate the base rate information to degree $\lambda\in[0,1]$, with
$\lambda=0$ indicating complete ignorance and $\lambda=1$ perfect Bayesian
updating. To evaluate aggregators' performance, we adopt Arieli et al. (2018)'s
worst-case regret model, which measures the maximum regret across the set of
considered information structures compared to an omniscient benchmark. Our
results reveal the surprising V-shape of regret as a function of $\lambda$.
That is, predictions with an intermediate incorporating degree of base rate
$\lambda<1$ can counter-intuitively lead to lower regret than perfect Bayesian
posteriors with $\lambda=1$. We additionally propose a new aggregator with low
regret robust to unknown $\lambda$. Finally, we conduct an empirical study to
test the base rate neglect model and evaluate the performance of various
aggregators.

我们研究了预测聚合中考虑基本概率忽视的问题，发现中等程度的基本概率忽视可以降低后悔程度，在最坏情况后悔模型中表现出 V 形曲线。我们还提出了一种在未知基本概率的情况下具有低后悔的新的聚合方法，并进行了实证研究来测试基本概率忽视模型和不同聚合方法的性能。

鲁棒聚合中基本率忽视的惊人益处

The Surprising Benefits of Base Rate Neglect in Robust Aggregation

Human forecasting accuracy in practice relies on the 'wisdom of the crowd'
effect, in which predictions about future events are significantly improved by
aggregating across a crowd of individual forecasters. Past work on the
forecasting ability of large language models (LLMs) suggests that frontier
LLMs, as individual forecasters, underperform compared to the gold standard of
a human crowd forecasting tournament aggregate. In Study 1, we expand this
research by using an LLM ensemble approach consisting of a crowd of twelve
LLMs. We compare the aggregated LLM predictions on 31 binary questions to that
of a crowd of 925 human forecasters from a three-month forecasting tournament.
Our main analysis shows that the LLM crowd outperforms a simple no-information
benchmark and is statistically equivalent to the human crowd. We also observe
an acquiescence effect, with mean model predictions being significantly above
50%, despite an almost even split of positive and negative resolutions.
Moreover, in Study 2, we test whether LLM predictions (of GPT-4 and Claude 2)
can be improved by drawing on human cognitive output. We find that both models'
forecasting accuracy benefits from exposure to the median human prediction as
information, improving accuracy by between 17% and 28%: though this leads to
less accurate predictions than simply averaging human and machine forecasts.
Our results suggest that LLMs can achieve forecasting accuracy rivaling that of
human crowd forecasting tournaments: via the simple, practically applicable
method of forecast aggregation. This replicates the 'wisdom of the crowd'
effect for LLMs, and opens up their use for a variety applications throughout
society.

人类预测准确性依赖于 “群体智慧” 效应，即通过对一群个体预测者进行聚合，可以显著提高对未来事件的预测。本研究通过使用由 12 个大型语言模型（LLM）组成的群体，扩展了过去关于大型语言模型作为预测者的研究。结果表明，通过简单且实际可行的预测聚合方法，LLMs 能够获得与人类群体预测锦标赛相媲美的准确性，从而复制了 “群体智慧” 效应，为 LLMs 在社会各个领域的应用开辟了可能。

硅众智慧：LLM 集合预测能力能媲美人工智能准确性

Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Match  Human Crowd Accuracy

Forecast aggregation combines the predictions of multiple forecasters to
improve accuracy. However, the lack of knowledge about forecasters' information
structure hinders optimal aggregation. Given a family of information
structures, robust forecast aggregation aims to find the aggregator with
minimal worst-case regret compared to the omniscient aggregator. Previous
approaches for robust forecast aggregation rely on heuristic observations and
parameter tuning. We propose an algorithmic framework for robust forecast
aggregation. Our framework provides efficient approximation schemes for general
information aggregation with a finite family of possible information
structures. In the setting considered by Arieli et al. (2018) where two agents
receive independent signals conditioned on a binary state, our framework also
provides efficient approximation schemes by imposing Lipschitz conditions on
the aggregator or discrete conditions on agents' reports. Numerical experiments
demonstrate the effectiveness of our method by providing a nearly optimal
aggregator in the setting considered by Arieli et al. (2018).

通过建立一个算法框架，我们提供了一种高效的逼近方案，用于在具有有限信息结构的情况下进行预测聚合，同时考虑了 Lipschitz 条件和离散条件。数值实验表明，我们的方法在所考虑的情景中能够提供几乎最优的聚合器。