Advances in deep learning systems have allowed large models to match or
surpass human accuracy on a number of skills such as image classification,
basic programming, and standardized test taking. As the performance of the most
capable models begin to saturate on tasks where humans already achieve high
accuracy, it becomes necessary to benchmark models on increasingly complex
abilities. One such task is forecasting the future outcome of events. In this
work we describe experiments using a novel dataset of real world events and
associated human predictions, an evaluation metric to measure forecasting
ability, and the accuracy of a number of different LLM based forecasting
designs on the provided dataset. Additionally, we analyze the performance of
the LLM forecasters against human predictions and find that models still
struggle to make accurate predictions about the future. Our follow-up
experiments indicate this is likely due to models' tendency to guess that most
events are unlikely to occur (which tends to be true for many prediction
datasets, but does not reflect actual forecasting abilities). We reflect on
next steps for developing a systematic and reliable approach to studying LLM
forecasting.

深度学习系统在图像分类、基本编程和标准化测试中已经达到或超过了人类的准确率。然而，在预测未来事件的能力上，模型仍然难以取得准确的结果。本研究使用了一种新颖的真实世界事件数据集和相关的人类预测，提出了一个评估指标来衡量预测能力，并对不同基于 LLM 的预测模型在该数据集上的准确性进行了分析。结果显示，模型在预测未来方面仍然存在困难，可能是由于模型倾向于猜测大多数事件不太可能发生，这在许多预测数据集中是正确的，但并不反映实际的预测能力。我们对开发系统化和可靠的 LLM 预测方法的下一步工作进行了思考。

语言模型是否能使用预测策略？

Can Language Models Use Forecasting Strategies?

Are the predictions of humans and language models affected by similar things?
Research suggests that while comprehending language, humans make predictions
about upcoming words, with more predictable words being processed more easily.
However, evidence also shows that humans display a similar processing advantage
for highly anomalous words when these words are semantically related to the
preceding context or to the most probable continuation. Using stimuli from 3
psycholinguistic experiments, we find that this is also almost always also the
case for 8 contemporary transformer language models (BERT, ALBERT, RoBERTa,
XLM-R, GPT-2, GPT-Neo, GPT-J, and XGLM). We then discuss the implications of
this phenomenon for our understanding of both human language comprehension and
the predictions made by language models.

使用来自 3 个心理语言学实验的刺激，发现当高度异常的单词与前面的语境或最有可能的延续相关时，8 个现代转换器语言模型（BERT、ALBERT、RoBERTa、XLM-R、GPT-2、GPT-Neo、GPT-J 和 XGLM）也几乎总是表现出类似的加工优势，这个现象对于我们对于人类语言理解和语言模型预测的理解都有重要意义。