In this paper, we propose sequence-based pretraining methods to enhance
procedural understanding in natural language processing. Procedural text,
containing sequential instructions to accomplish a task, is difficult to
understand due to the changing attributes of entities in the context. We focus
on recipes, which are commonly represented as ordered instructions, and use
this order as a supervision signal. Our work is one of the first to compare
several 'order as-supervision' transformer pre-training methods, including
Permutation Classification, Embedding Regression, and Skip-Clip, and shows that
these methods give improved results compared to the baselines and SoTA LLMs on
two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain
and ProPara dataset in open domain. Our proposed methods address the
non-trivial Entity Tracking Task that requires prediction of entity states
across procedure steps, which requires understanding the order of steps. These
methods show an improvement over the best baseline by 1.6% and 7-9% on
NPN-Cooking and ProPara Datasets respectively across metrics.

我们提出了一种基于序列的预训练方法，以加强自然语言处理中的程序理解。我们的工作是首次比较了几种 “以顺序为监督” 的 Transformer 预训练方法，并显示这些方法在两个下游实体跟踪数据集（食谱领域的 NPN-Cooking 数据集和开放领域的 ProPara 数据集）上相对于基准模型和最新方法有了改进的结果。我们的方法解决了需要预测过程步骤中实体状态的非平凡的实体跟踪任务，这需要理解步骤的顺序。这些方法在 NPN-Cooking 和 ProPara 数据集上相对于最佳基准模型分别在度量指标上提高了 1.6％和 7-9％。

基于顺序的程序化文本理解的预训练策略

Order-Based Pre-training Strategies for Procedural Text Understanding

Stories about everyday situations are an essential part of human
communication, motivating the need to develop AI agents that can reliably
understand these stories. Despite the long list of supervised methods for story
completion and procedural understanding, current AI has no mechanisms to
automatically track and explain procedures in unseen stories. To bridge this
gap, we study the ability of AI models to transfer procedural knowledge to
novel narrative tasks in a transparent manner. We design LEAP: a comprehensive
framework that integrates state-of-the-art modeling architectures, training
regimes, and augmentation strategies based on both natural and synthetic
stories. To address the lack of densely annotated training data, we devise a
robust automatic labeler based on few-shot prompting to enhance the augmented
data. Our experiments with in- and out-of-domain tasks reveal insights into the
interplay of different architectures, training regimes, and augmentation
strategies. LEAP's labeler has a clear positive impact on out-of-domain
datasets, while the resulting dense annotation provides native explainability.

研究了 AI 模型透明地将过程知识传输到新的叙述任务的能力，并设计了 LEAP 框架，该框架基于自然和合成故事的一流建模体系结构、训练方案和增强策略，包括基于少量提示的强健自动标签器，来增强受到稀缺注释训练数据的学习。