Keeping track of how states and relations of entities change as a text or
dialog unfolds is a key prerequisite to discourse understanding. Despite this
fact, there have been few systematic investigations into the ability of large
language models (LLMs) to track discourse entities. In this work, we present a
task to probe to what extent a language model can infer the final state of an
entity given an English description of the initial state and a series of
state-changing operations. We use this task to first investigate whether
Flan-T5, GPT-3 and GPT-3.5 can track the state of entities, and find that only
GPT-3.5 models, which have been pretrained on large amounts of code, exhibit
this ability. We then investigate whether smaller models pretrained primarily
on text can learn to track entities, through finetuning T5 on several
training/evaluation splits. While performance degrades for more complex splits,
we find that even for splits with almost no lexical overlap between training
and evaluation, a finetuned model can often perform non-trivial entity
tracking. Taken together, these results suggest that language models can learn
to track entities but pretraining on large text corpora alone does not make
this capacity surface.

本文探讨了大语言模型在跟踪实体状态和关系变化方面的能力，发现只有预训练于大量代码的 GPT-3.5 模型具有此能力，而使用预训练于文本的较小模型进行微调后也可以完成一定程度的实体追踪。但这种能力不仅取决于模型的大小，大文本库的预训练也不是必要条件。