Forecasting future world events is a challenging but valuable task. Forecasts
of climate, geopolitical conflict, pandemics and economic indicators help shape
policy and decision making. In these domains, the judgment of expert humans
contributes to the best forecasts. Given advances in language modeling, can
these forecasts be automated? To this end, we introduce Autocast, a dataset
containing thousands of forecasting questions and an accompanying news corpus.
Questions are taken from forecasting tournaments, ensuring high quality,
real-world importance, and diversity. The news corpus is organized by date,
allowing us to precisely simulate the conditions under which humans made past
forecasts (avoiding leakage from the future). Motivated by the difficulty of
forecasting numbers across orders of magnitude (e.g. global cases of COVID-19
in 2022), we also curate IntervalQA, a dataset of numerical questions and
metrics for calibration. We test language models on our forecasting task and
find that performance is far below a human expert baseline. However,
performance improves with increased model size and incorporation of relevant
information from the news corpus. In sum, Autocast poses a novel challenge for
large language models and improved performance could bring large practical
benefits.

该研究介绍了 Autocast 数据集以及伴随的新闻语料库，以提高语言模型的预测能力，其中还包括数字问题和度量标准的 IntervalQA 数据集，并发现语言模型的性能远低于人类专家基准，但随着模型规模和新闻语料库相关信息的增加，性能有所提高。