We propose a visually grounded speech model that acquires new words and their
visual depictions from just a few word-image example pairs. Given a set of test
images and a spoken query, we ask the model which image depicts the query word.
Previous work has simplified this problem by either using an artificial setting
with digit word-image pairs or by using a large number of examples per class.
We propose an approach that can work on natural word-image pairs but with less
examples, i.e. fewer shots. Our approach involves using the given word-image
example pairs to mine new unsupervised word-image training pairs from large
collections of unlabelled speech and images. Additionally, we use a
word-to-image attention mechanism to determine word-image similarity. With this
new model, we achieve better performance with fewer shots than any existing
approach.

本文提出了一个视觉和语音相融合的模型，用于从仅有几个词 - 图像样本对中学习新单词及其视觉表示。我们的方法包括从大量未标记的语音和图像中，使用给定的词 - 图像示例对挖掘新的无监督词 - 图像训练对，并使用单词到图像的关注机制来确定词 - 图像相似性。新模型的性能比现有方法更好，且需要更少的样本数。