We present an approach to improve statistical machine translation of image
descriptions by multimodal pivots defined in visual space. The key idea is to
perform image retrieval over a database of images that are captioned in the
target language, and use the captions of the most similar images for
crosslingual reranking of translation outputs. Our approach does not depend on
the availability of large amounts of in-domain parallel data, but only relies
on available large datasets of monolingually captioned images, and on
state-of-the-art convolutional neural networks to compute image similarities.
Our experimental evaluation shows improvements of 1 BLEU point over strong
baselines.

本文提供了一种使用视觉空间中的多模态支点改善图像描述的统计机器翻译方法。通过在一个用目标语言描述的图像数据库中执行图像检索，并使用最相似图像的描述进行跨语言重新排序，达到优化的效果。本方法不依赖于大量域内平行数据的可用性，而仅依赖于大量单语言字幕图像数据集的可用性，以及用于计算图像相似性的最先进的卷积神经网络。我们的实验评估显示，与强基线相比，我们的方法提高了 1 个 BLEU 分数。