Understanding food recipe requires anticipating the implicit causal effects
of cooking actions, such that the recipe can be converted into a graph
describing the temporal workflow of the recipe. This is a non-trivial task that
involves common-sense reasoning. However, existing efforts rely on hand-crafted
features to extract the workflow graph from recipes due to the lack of
large-scale labeled datasets. Moreover, they fail to utilize the cooking
images, which constitute an important part of food recipes. In this paper, we
build MM-ReS, the first large-scale dataset for cooking workflow construction,
consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps
are multi-modal, featuring both text instructions and cooking images. We then
propose a neural encoder-decoder model that utilizes both visual and textual
information to construct the cooking workflow, which achieved over 20%
performance gain over existing hand-crafted baselines.

利用图形化方法描述菜谱的工作流程是一个需要常识推理的复杂任务。本文提出了一个基于神经网络技术的编码器 - 解码器模型，通过使用视觉和文本信息来构建食品制作工作流程。我们还构建了一个包含 9,850 道菜谱的大型数据集，采用该方法比手工特征提取能够显著提高 20% 以上的性能。