Cross-modal recipe retrieval has recently gained substantial attention due to
the importance of food in people's lives, as well as the availability of vast
amounts of digital cooking recipes and food images to train machine learning
models. In this work, we revisit existing approaches for cross-modal recipe
retrieval and propose a simplified end-to-end model based on well established
and high performing encoders for text and images. We introduce a hierarchical
recipe Transformer which attentively encodes individual recipe components
(titles, ingredients and instructions). Further, we propose a self-supervised
loss function computed on top of pairs of individual recipe components, which
is able to leverage semantic relationships within recipes, and enables training
using both image-recipe and recipe-only samples. We conduct a thorough analysis
and ablation studies to validate our design choices. As a result, our proposed
method achieves state-of-the-art performance in the cross-modal recipe
retrieval task on the Recipe1M dataset. We make code and models publicly
available.

本研究针对跨模态食谱检索任务，提出一个基于端到端模型的层级食谱 Transformer，使用自我监督损失函数在图像和食谱样本上训练，取得了 Recipe1M 数据集上的最佳性能表现。