In this paper, we propose CLMSM, a domain-specific, continual pre-training
framework, that learns from a large set of procedural recipes. CLMSM uses a
Multi-Task Learning Framework to optimize two objectives - a) Contrastive
Learning using hard triplets to learn fine-grained differences across entities
in the procedures, and b) a novel Mask-Step Modelling objective to learn
step-wise context of a procedure. We test the performance of CLMSM on the
downstream tasks of tracking entities and aligning actions between two
procedures on three datasets, one of which is an open-domain dataset not
conforming with the pre-training dataset. We show that CLMSM not only
outperforms baselines on recipes (in-domain) but is also able to generalize to
open-domain procedural NLP tasks.

提出了 CLMSM：一种针对特定领域的连续预训练框架，通过学习大量过程配方来优化两个目标，一是使用硬三元组的对比学习来学习程序实体间的细粒度差异，二是使用新颖的遮蔽 - 步骤建模目标来学习步骤的上下文。通过在三个数据集上进行实体跟踪和动作对齐的下游任务上测试了 CLMSM 的性能，其中一个数据集是不符合预训练数据集的开放域数据集。结果表明，CLMSM 不仅在配方（领域内）上优于基准模型，而且能够推广到开放域过程性自然语言处理任务。