The objective of image captioning models is to bridge the gap between the
visual and linguistic modalities by generating natural language descriptions
that accurately reflect the content of input images. In recent years,
researchers have leveraged deep learning-based models and made advances in the
extraction of visual features and the design of multimodal connections to
tackle this task. This work presents a novel approach towards developing image
captioning models that utilize an external kNN memory to improve the generation
process. Specifically, we propose two model variants that incorporate a
knowledge retriever component that is based on visual similarities, a
differentiable encoder to represent input images, and a kNN-augmented language
model to predict tokens based on contextual cues and text retrieved from the
external memory. We experimentally validate our approach on COCO and nocaps
datasets and demonstrate that incorporating an explicit external memory can
significantly enhance the quality of captions, especially with a larger
retrieval corpus. This work provides valuable insights into retrieval-augmented
captioning models and opens up new avenues for improving image captioning at a
larger scale.

通过利用外部的 kNN 内存来改善生成过程，本研究提出了两个模型变体，这些模型变体结合了基于视觉相似性的知识检索器组件、可微分编码器以及基于上下文线索和从外部存储器中检索的文本来预测标记的 kNN 增强语言模型。在 COCO 和 nocaps 数据集上的实验证实了我们的方法，证明了显式外部存储器的纳入可以显著提高标题的质量，尤其是在更大的检索语料库中。此研究为改善大规模的图像字幕生成打开了新的研究方向。

面向图像标注的检索增强架构

Towards Retrieval-Augmented Architectures for Image Captioning

Image captioning models aim at connecting Vision and Language by providing
natural language descriptions of input images. In the past few years, the task
has been tackled by learning parametric models and proposing visual feature
extraction advancements or by modeling better multi-modal connections. In this
paper, we investigate the development of an image captioning approach with a
kNN memory, with which knowledge can be retrieved from an external corpus to
aid the generation process. Our architecture combines a knowledge retriever
based on visual similarities, a differentiable encoder, and a kNN-augmented
attention layer to predict tokens based on the past context and on text
retrieved from the external memory. Experimental results, conducted on the COCO
dataset, demonstrate that employing an explicit external memory can aid the
generation process and increase caption quality. Our work opens up new avenues
for improving image captioning models at larger scale.

本论文中，我们研究了一种基于 kNN 记忆的图像字幕生成方法，其中可以从外部语料库中检索知识来辅助生成过程。通过在视觉相似性、差分编码器和 kNN 增强的注意力层之间结合知识检索器来预测基于上下文和从外部内存检索的文本的令牌。在 COCO 数据集上进行的实验结果表明，采用显式的外部存储器可以帮助生成过程并提高字幕质量。我们的工作为更大规模的图像字幕生成模型的改进开辟了新的途径。