Words of estimative probability (WEPs), such as ''maybe'' or ''probably not''
are ubiquitous in natural language for communicating estimative uncertainty,
compared with direct statements involving numerical probability. Human
estimative uncertainty, and its calibration with numerical estimates, has long
been an area of study -- including by intelligence agencies like the CIA. This
study compares estimative uncertainty in commonly used large language models
(LLMs) like GPT-4 and ERNIE-4 to that of humans, and to each other. Here we
show that LLMs like GPT-3.5 and GPT-4 align with human estimates for some, but
not all, WEPs presented in English. Divergence is also observed when the LLM is
presented with gendered roles and Chinese contexts. Further study shows that an
advanced LLM like GPT-4 can consistently map between statistical and estimative
uncertainty, but a significant performance gap remains. The results contribute
to a growing body of research on human-LLM alignment.

用统计的方法对 GPT-4 等大型语言模型进行估测不确定性的研究发现，虽然 GPT-4 在一些英文词组中与人类的估测结果接近，但在涉及到性别角色和中文语境时存在差异，证明人类与大型语言模型的估测结果尚有较大难以弥合的差距。