Are Large language models (LLMs) temporally grounded? Since LLMs cannot
perceive and interact with the environment, it is impossible to answer this
question directly. Instead, we provide LLMs with textual narratives and probe
them with respect to their common-sense knowledge of the structure and duration
of events, their ability to order events along a timeline, and self-consistency
within their temporal model (e.g., temporal relations such as after and before
are mutually exclusive for any pair of events). We evaluate state-of-the-art
LLMs (such as LLaMA 2 and GPT-4) on three tasks reflecting these abilities.
Generally, we find that LLMs lag significantly behind both human performance as
well as small-scale, specialised LMs. In-context learning, instruction tuning,
and chain-of-thought prompting reduce this gap only to a limited degree.
Crucially, LLMs struggle the most with self-consistency, displaying incoherent
behaviour in at least 27.23% of their predictions. Contrary to expectations, we
also find that scaling the model size does not guarantee positive gains in
performance. To explain these results, we study the sources from which LLMs may
gather temporal information: we find that sentence ordering in unlabelled
texts, available during pre-training, is only weakly correlated with event
ordering. Moreover, public instruction tuning mixtures contain few temporal
tasks. Hence, we conclude that current LLMs lack a consistent temporal model of
textual narratives. Code, datasets, and LLM outputs are available at
this https URL

大型语言模型在时间上没有实质依据，缺乏自洽性并且表现出不一致的行为性。