Nowadays, driven by the increasing concern on diet and health, food computing
has attracted enormous attention from both industry and research community. One
of the most popular research topics in this domain is Food Retrieval, due to
its profound influence on health-oriented applications. In this paper, we focus
on the task of cross-modal retrieval between food images and cooking recipes.
We present Modality-Consistent Embedding Network (MCEN) that learns
modality-invariant representations by projecting images and texts to the same
embedding space. To capture the latent alignments between modalities, we
incorporate stochastic latent variables to explicitly exploit the interactions
between textual and visual features. Importantly, our method learns the
cross-modal alignments during training but computes embeddings of different
modalities independently at inference time for the sake of efficiency.
Extensive experimental results clearly demonstrate that the proposed MCEN
outperforms all existing approaches on the benchmark Recipe1M dataset and
requires less computational cost.

本文介绍一种通过 “模态一致嵌入网络” 实现食品图片与烹饪配方的跨模态检索的方法，旨在提升健康应用领域中的食品检索技术。经过广泛的实验结果证明，该方法在 Recipe1M 数据集上表现优秀并且需要较少的计算成本。

利用潜变量模型拉近烹饪食谱和菜肴图像之间的跨模态差距

MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images  with Latent Variable Model

Food retrieval is an important task to perform analysis of food-related
information, where we are interested in retrieving relevant information about
the queried food item such as ingredients, cooking instructions, etc. In this
paper, we investigate cross-modal retrieval between food images and cooking
recipes. The goal is to learn an embedding of images and recipes in a common
feature space, such that the corresponding image-recipe embeddings lie close to
one another. Two major challenges in addressing this problem are 1) large
intra-variance and small inter-variance across cross-modal food data; and 2)
difficulties in obtaining discriminative recipe representations. To address
these two problems, we propose Semantic-Consistent and Attention-based Networks
(SCAN), which regularize the embeddings of the two modalities through aligning
output semantic probabilities. Besides, we exploit a self-attention mechanism
to improve the embedding of recipes. We evaluate the performance of the
proposed method on the large-scale Recipe1M dataset, and show that we can
outperform several state-of-the-art cross-modal retrieval strategies for food
images and cooking recipes by a significant margin.

本文提出了一种名为 Semantic-Consistent and Attention-based Networks (SCAN) 的方法，可解决在跨模式食品数据中存在的内在方差大和间方差小的问题以及获取区分性食谱表示的难点，其目的是学习图像和食谱的嵌入，使相应的图像 - 食谱嵌入彼此靠近，然后在大规模的 Recipe1M 数据集上评估了该方法的性能，并表明它在食品图像和烹饪食谱的交叉检索方面优于几种最先进的策略。