Despite the abundance of multi-modal data, such as image-text pairs, there
has been little effort in understanding the individual entities and their
different roles in the construction of these data instances. In this work, we
endeavour to discover the entities and their corresponding importance in
cooking recipes automaticall} as a visual-linguistic association problem. More
specifically, we introduce a novel cross-modal learning framework to jointly
model the latent representations of images and text in the food image-recipe
association and retrieval tasks. This model allows one to discover complex
functional and hierarchical relationships between images and text, and among
textual parts of a recipe including title, ingredients and cooking
instructions. Our experiments show that by making use of efficient
tree-structured Long Short-Term Memory as the text encoder in our computational
cross-modal retrieval framework, we are not only able to identify the main
ingredients and cooking actions in the recipe descriptions without explicit
supervision, but we can also learn more meaningful feature representations of
food recipes, appropriate for challenging cross-modal retrieval and recipe
adaption tasks.

本文提出了一种新颖的跨模态学习框架来自动发现食谱中文本和图像的实体及其相应的重要性，并能在没有明确监督的情况下识别出食谱描述中的主要成分和烹饪动作，为跨模态检索和食谱适应任务提供更有意义的特征表示。