In this paper, we are interested in modeling a how-to instructional
procedure, such as a cooking recipe, with a meaningful and rich high-level
representation. Specifically, we propose to represent cooking recipes and food
images as cooking programs. Programs provide a structured representation of the
task, capturing cooking semantics and sequential relationships of actions in
the form of a graph. This allows them to be easily manipulated by users and
executed by agents. To this end, we build a model that is trained to learn a
joint embedding between recipes and food images via self-supervision and
jointly generate a program from this embedding as a sequence. To validate our
idea, we crowdsource programs for cooking recipes and show that: (a) projecting
the image-recipe embeddings into programs leads to better cross-modal retrieval
results; (b) generating programs from images leads to better recognition
results compared to predicting raw cooking instructions; and (c) we can
generate food images by manipulating programs via optimizing the latent code of
a GAN. Code, data, and models are available online.

本文提出了一种通过将烹饪配方和食品图像表示为烹饪程序来建模如何指导说明性程序（如烹饪食谱），并展示了使用自我监督和联合嵌入生成程序的可行性，从图像生成食品图像，这使得跨模态检索效果更佳。