The objective of image captioning models is to bridge the gap between the
visual and linguistic modalities by generating natural language descriptions
that accurately reflect the content of input images. In recent years,
researchers have leveraged deep learning-based models and made advances in the
extraction of visual features and the design of multimodal connections to
tackle this task. This work presents a novel approach towards developing image
captioning models that utilize an external kNN memory to improve the generation
process. Specifically, we propose two model variants that incorporate a
knowledge retriever component that is based on visual similarities, a
differentiable encoder to represent input images, and a kNN-augmented language
model to predict tokens based on contextual cues and text retrieved from the
external memory. We experimentally validate our approach on COCO and nocaps
datasets and demonstrate that incorporating an explicit external memory can
significantly enhance the quality of captions, especially with a larger
retrieval corpus. This work provides valuable insights into retrieval-augmented
captioning models and opens up new avenues for improving image captioning at a
larger scale.

通过利用外部的 kNN 内存来改善生成过程，本研究提出了两个模型变体，这些模型变体结合了基于视觉相似性的知识检索器组件、可微分编码器以及基于上下文线索和从外部存储器中检索的文本来预测标记的 kNN 增强语言模型。在 COCO 和 nocaps 数据集上的实验证实了我们的方法，证明了显式外部存储器的纳入可以显著提高标题的质量，尤其是在更大的检索语料库中。此研究为改善大规模的图像字幕生成打开了新的研究方向。