Uncertainty expressions such as ``probably'' or ``highly unlikely'' are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans interpret these expressions, there has been little inquiry into the abilities of language models to interpret such expressions. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We evaluate both humans and 10 popular language models on a task created to assess these abilities. Unexpectedly, we find that 8 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI alignment and AI-AI communication.

本研究解决了语言模型在理解不确定性表达（如“可能”或“极不可能”）方面的能力缺陷。我们采用了理论思维的视角，评估语言模型在与人类进行不确定性表达解读时的表现。研究发现，尽管大多数语言模型能够以类似人类的方式映射不确定性表达成概率响应，但它们的反应在真假陈述上的偏差显示出它们更易受先前知识的影响，这对人机协作及AI间的沟通具有深远影响。

语言模型和人类对语言不确定性的认知