The ability to sequence unordered events is an essential skill to comprehend
and reason about real world task procedures, which often requires thorough
understanding of temporal common sense and multimodal information, as these
procedures are often communicated through a combination of texts and images.
Such capability is essential for applications such as sequential task planning
and multi-source instruction summarization. While humans are capable of
reasoning about and sequencing unordered multimodal procedural instructions,
whether current machine learning models have such essential capability is still
an open question. In this work, we benchmark models' capability of reasoning
over and sequencing unordered multimodal instructions by curating datasets from
popular online instructional manuals and collecting comprehensive human
annotations. We find models not only perform significantly worse than humans
but also seem incapable of efficiently utilizing the multimodal information. To
improve machines' performance on multimodal event sequencing, we propose
sequentiality-aware pretraining techniques that exploit the sequential
alignment properties of both texts and images, resulting in > 5% significant
improvements.

本文通过构建数据集和收集人类注释来检验机器学习模型分析和排序多模式事件的能力，发现机器模型不仅表现远远不如人类，而且不能有效地利用多模式信息。为了改善机器在多模式事件排序上的性能，作者提出了针对顺序性的预训练技术，可以获得显著的 > 5% 的性能提升。

通过对多模态指南进行排序来理解多模态程序化知识

Understanding Multimodal Procedural Knowledge by Sequencing Multimodal  Instructional Manuals

Temporal common sense (e.g., duration and frequency of events) is crucial for
understanding natural language. However, its acquisition is challenging, partly
because such information is often not expressed explicitly in text, and human
annotation on such concepts is costly. This work proposes a novel sequence
modeling approach that exploits explicit and implicit mentions of temporal
common sense, extracted from a large corpus, to build TACOLM, a temporal common
sense language model. Our method is shown to give quality predictions of
various dimensions of temporal common sense (on UDST and a newly collected
dataset from RealNews). It also produces representations of events for relevant
tasks such as duration comparison, parent-child relations, event coreference
and temporal QA (on TimeBank, HiEVE and MCTACO) that are better than using the
standard BERT. Thus, it will be an important component of temporal NLP.

本论文提出了一种基于序列建模的方法，利用大型语料库中显式和隐含的时间常识，构建了一个新颖的时间常识语言模型 TACOLM。该模型可以有效预测各种维度的时间常识，并且在时态 NLP 相关任务中产生了比标准 BERT 更好的事件表示。

最小监督的时间常识获取

Temporal Common Sense Acquisition with Minimal Supervision

Temporal common sense has applications in AI tasks such as QA, multi-document
summarization, and human-AI communication. We propose the task of sequencing --
given a jumbled set of aligned image-caption pairs that belong to a story, the
task is to sort them such that the output sequence forms a coherent story. We
present multiple approaches, via unary (position) and pairwise (order)
predictions, and their ensemble-based combinations, achieving strong results on
this task. We use both text-based and image-based features, which depict
complementary improvements. Using qualitative examples, we demonstrate that our
models have learnt interesting aspects of temporal common sense.

给定图像与描述文本，使用排序方法对其进行一致性的故事串联。通过多种方法设计组合模型，使用了基于文本和图像特征的成分。模型具备了学会了时间共识的有趣方面的能力。