Sharing food has become very popular with the development of social media.
For many real-world applications, people are keen to know the underlying
recipes of a food item. In this paper, we are interested in automatically
generating cooking instructions for food. We investigate an open research task
of generating cooking instructions based on only food images and ingredients,
which is similar to the image captioning task. However, compared with image
captioning datasets, the target recipes are long-length paragraphs and do not
have annotations on structure information. To address the above limitations, we
propose a novel framework of Structure-aware Generation Network (SGN) to tackle
the food recipe generation task. Our approach brings together several novel
ideas in a systematic framework: (1) exploiting an unsupervised learning
approach to obtain the sentence-level tree structure labels before training;
(2) generating trees of target recipes from images with the supervision of tree
structure labels learned from (1); and (3) integrating the inferred tree
structures with the recipe generation procedure. Our proposed model can produce
high-quality and coherent recipes, and achieve the state-of-the-art performance
on the benchmark Recipe1M dataset.

本研究提出了一个单纯使用食品图片和配料来生成食品烹饪说明的结构感知生成网络，通过无监督学习的方式，提取句子级别的树状结构标签，并将其与生成的目标食谱树形结构相结合。实验表明该方法能产生高质量连贯的食谱，而且在基准的 Recipe1M 数据集上取得了最先进的性能。