Cross-modal image-recipe retrieval has gained significant attention in recent years. Most work focuses on improving cross-modal embeddings using unimodal encoders, that allow for efficient retrieval in large-scale databases, leaving aside cross-attention between modalities which is more computationally expensive. We propose a new retrieval framework, T-Food (Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval) that exploits the interaction between modalities in a novel regularization scheme, while using only unimodal encoders at test time for efficient retrieval. We also capture the intra-dependencies between recipe entities with a dedicated recipe encoder, and propose new variants of triplet losses with dynamic margins that adapt to the difficulty of the task. Finally, we leverage the power of the recent Vision and Language Pretraining (VLP) models such as CLIP for the image encoder. Our approach outperforms existing approaches by a large margin on the Recipe1M dataset. Specifically, we achieve absolute improvements of 8.1 % (72.6 R@1) and +10.9 % (44.6 R@1) on the 1k and 10k test sets respectively. The code is available here:https://github.com/mshukor/TFood

本研究提出了一种名为T-Food的跨媒体图像-食谱检索框架，利用多模态正则化技术，从而在测试时使用单模态编码器进行高效检索，可以捕捉食谱实体之间的内部依赖关系，并提出了动态边缘的三元组损失变体。最后，本文还利用最新的视觉与语言预训练（VLP）模型，如CLIP，作为图像编码器，并在Recipe1M数据集上实现了比现有方法大幅提升的结果。

跨模态食物检索的多模态正则化Transformer解码器