Understanding food recipe requires anticipating the implicit causal effects
of cooking actions, such that the recipe can be converted into a graph
describing the temporal workflow of the recipe. This is a non-trivial task that
involves common-sense reasoning. However, existing efforts rely on hand-crafted
features to extract the workflow graph from recipes due to the lack of
large-scale labeled datasets. Moreover, they fail to utilize the cooking
images, which constitute an important part of food recipes. In this paper, we
build MM-ReS, the first large-scale dataset for cooking workflow construction,
consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps
are multi-modal, featuring both text instructions and cooking images. We then
propose a neural encoder-decoder model that utilizes both visual and textual
information to construct the cooking workflow, which achieved over 20%
performance gain over existing hand-crafted baselines.

利用图形化方法描述菜谱的工作流程是一个需要常识推理的复杂任务。本文提出了一个基于神经网络技术的编码器 - 解码器模型，通过使用视觉和文本信息来构建食品制作工作流程。我们还构建了一个包含 9,850 道菜谱的大型数据集，采用该方法比手工特征提取能够显著提高 20% 以上的性能。

多模式烹饪工作流程构建用于食谱

Multi-modal Cooking Workflow Construction for Food Recipes

A food recipe is an ordered set of instructions for preparing a particular
dish. From a visual perspective, every instruction step can be seen as a way to
change the visual appearance of the dish by adding extra objects (e.g., adding
an ingredient) or changing the appearance of the existing ones (e.g., cooking
the dish). In this paper, we aim to teach a machine how to make a pizza by
building a generative model that mirrors this step-by-step procedure. To do so,
we learn composable module operations which are able to either add or remove a
particular ingredient. Each operator is designed as a Generative Adversarial
Network (GAN). Given only weak image-level supervision, the operators are
trained to generate a visual layer that needs to be added to or removed from
the existing image. The proposed model is able to decompose an image into an
ordered sequence of layers by applying sequentially in the right order the
corresponding removing modules. Experimental results on synthetic and real
pizza images demonstrate that our proposed model is able to: (1) segment pizza
toppings in a weaklysupervised fashion, (2) remove them by revealing what is
occluded underneath them (i.e., inpainting), and (3) infer the ordering of the
toppings without any depth ordering supervision. Code, data, and models are
available online.

本文提出了一种基于生成对抗网络（GAN）的可组合模块运算的生成模型，可以将图像分解为一系列层，从而实现在无监督的情况下分割食物、补全图像和推断配菜的顺序。