Tracking entities in procedural language requires understanding the
transformations arising from actions on entities as well as those entities'
interactions. While self-attention-based pre-trained language encoders like GPT
and BERT have been successfully applied across a range of natural language
understanding tasks, their ability to handle the nuances of procedural texts is
still untested. In this paper, we explore the use of pre-trained transformer
networks for entity tracking tasks in procedural text. First, we test standard
lightweight approaches for prediction with pre-trained transformers, and find
that these approaches underperform even simple baselines. We show that much
stronger results can be attained by restructuring the input to guide the
transformer model to focus on a particular entity. Second, we assess the degree
to which transformer networks capture the process dynamics, investigating such
factors as merged entities and oblique entity references. On two different
tasks, ingredient detection in recipes and QA over scientific processes, we
achieve state-of-the-art results, but our models still largely attend to
shallow context clues and do not form complex representations of intermediate
entity or process state.

本研究探讨了在过程性文本中使用预训练的 transformer 网络跟踪实体的能力，发现通过调整输入结构，可以取得更好的实体跟踪结果，在配方中实现了材料检测以及在科学过程中进行了问答，达到了最先进的结果，但我们的模型仍然主要关注表面上下文线索，没有形成中间实体或进程状态的复杂表示。