In this paper, we present a cross-modal recipe retrieval framework,
Transformer-based Network for Large Batch Training (TNLBT), which is inspired
by ACME~(Adversarial Cross-Modal Embedding) and H-T~(Hierarchical Transformer).
TNLBT aims to accomplish retrieval tasks while generating images from recipe
embeddings. We apply the Hierarchical Transformer-based recipe text encoder,
the Vision Transformer~(ViT)-based recipe image encoder, and an adversarial
network architecture to enable better cross-modal embedding learning for recipe
texts and images. In addition, we use self-supervised learning to exploit the
rich information in the recipe texts having no corresponding images. Since
contrastive learning could benefit from a larger batch size according to the
recent literature on self-supervised learning, we adopt a large batch size
during training and have validated its effectiveness. In the experiments, the
proposed framework significantly outperformed the current state-of-the-art
frameworks in both cross-modal recipe retrieval and image generation tasks on
the benchmark Recipe1M. This is the first work which confirmed the
effectiveness of large batch training on cross-modal recipe embeddings.

本文提出了基于 Transformer 神经网络的大批量训练 (TNLBT) 的跨模型食谱检索框架，并应用层次 Transformer 的食谱文本编码器、Vision Transformer 的食谱图像编码器和对抗网络架构来实现更好的食谱文本和图像的交叉嵌入学习。同时利用自监督学习来挖掘不具备相应图像的食谱文本中的丰富信息，结果验证了使用大批量训练对跨模型食谱嵌入的有效性。在实验中，该框架显著优于当前的最先进跨模型食谱检索框架。

使用 Transformer 进行跨模态食谱嵌入及大批量训练

Transformer-based Cross-Modal Recipe Embeddings with Large Batch  Training

Cross-modal recipe retrieval has recently gained substantial attention due to
the importance of food in people's lives, as well as the availability of vast
amounts of digital cooking recipes and food images to train machine learning
models. In this work, we revisit existing approaches for cross-modal recipe
retrieval and propose a simplified end-to-end model based on well established
and high performing encoders for text and images. We introduce a hierarchical
recipe Transformer which attentively encodes individual recipe components
(titles, ingredients and instructions). Further, we propose a self-supervised
loss function computed on top of pairs of individual recipe components, which
is able to leverage semantic relationships within recipes, and enables training
using both image-recipe and recipe-only samples. We conduct a thorough analysis
and ablation studies to validate our design choices. As a result, our proposed
method achieves state-of-the-art performance in the cross-modal recipe
retrieval task on the Recipe1M dataset. We make code and models publicly
available.

本研究针对跨模态食谱检索任务，提出一个基于端到端模型的层级食谱 Transformer，使用自我监督损失函数在图像和食谱样本上训练，取得了 Recipe1M 数据集上的最佳性能表现。

层次变换器与自监督学习优化跨模态菜谱检索

Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers  and Self-supervised Learning

We propose a novel non-parametric method for cross-modal recipe retrieval
which is applied on top of precomputed image and text embeddings. By combining
our method with standard approaches for building image and text encoders,
trained independently with a self-supervised classification objective, we
create a baseline model which outperforms most existing methods on a
challenging image-to-recipe task. We also use our method for comparing image
and text encoders trained using different modern approaches, thus addressing
the issues hindering the development of novel methods for cross-modal recipe
retrieval. We demonstrate how to use the insights from model comparison and
extend our baseline model with standard triplet loss that improves
state-of-the-art on the Recipe1M dataset by a large margin, while using only
precomputed features and with much less complexity than existing methods.
Further, our approach readily generalizes beyond recipe retrieval to other
challenging domains, achieving state-of-the-art performance on Politics and
GoodNews cross-modal retrieval tasks.

我们提出了一种新颖的非参数方法，用于跨模式菜谱检索，结合图像和文本嵌入，通过将我们的方法与用自监督分类目标独立训练的标准方法相结合，我们创建了一个基准模型，在挑战性的图像到菜谱任务上优于大多数现有方法。我们还使用我们的方法比较使用不同现代方法训练的图像和文本编码器，从而解决跨模式菜谱检索的问题。通过三元组损失增强基准模型，同时仅使用预计算特征并且比现有方法更简单，大幅提高了在 Recipe1M 数据集上的最新水平，并且我们的方法易于推广到其他具有挑战性的领域，在政治和 GoodNews 跨模态检索任务上实现了最先进的性能。