Visually grounded speech models link speech to images. We extend this
connection by linking images to text via an existing image captioning system,
and as a result gain the ability to map speech audio directly to text. This
approach can be used for speech translation with just images by having the
audio in a different language from the generated captions. We investigate such
a system on a real low-resource language, Yor\`ub\'a, and propose a
Yor\`ub\'a-to-English speech translation model that leverages pretrained
components in order to be able to learn in the low-resource regime. To limit
overfitting, we find that it is essential to use a decoding scheme that
produces diverse image captions for training. Results show that the predicted
translations capture the main semantics of the spoken audio, albeit in a
simpler and shorter form.

通过现有的图像字幕系统，将图像与文本相连接，实现语音音频与文本之间的直接映射，从而扩展了视觉语音模型的连接方式。本研究在一个低资源语言 Yorùbá 上，提出了一种 Yorùbá- 英语语音翻译模型，利用预训练组件以实现在低资源环境中的学习，并通过使用能产生多样的图像字幕的解码方案来限制过拟合现象。结果显示预测的翻译捕捉到了口头音频的主要语义，尽管形式上更简单且更简短。