Multimodal machine translation (MMT) aims to improve neural machine
translation (NMT) with additional visual information, but most existing MMT
methods require paired input of source sentence and image, which makes them
suffer from shortage of sentence-image pairs. In this paper, we propose a
phrase-level retrieval-based method for MMT to get visual information for the
source input from existing sentence-image data sets so that MMT can break the
limitation of paired sentence-image input. Our method performs retrieval at the
phrase level and hence learns visual information from pairs of source phrase
and grounded region, which can mitigate data sparsity. Furthermore, our method
employs the conditional variational auto-encoder to learn visual
representations which can filter redundant visual information and only retain
visual information related to the phrase. Experiments show that the proposed
method significantly outperforms strong baselines on multiple MMT datasets,
especially when the textual context is limited.

对比于其它多模式机器翻译 (MMT) 方法，我们提出了一种短语级别检索的 MMT 方法，从已有的文本 - 图像数据集中获得源输入的视觉信息，有利于缓解输入限制和数据稀疏的问题，并且通过条件变分自编码器可以更好地过滤多余的视觉信息和仅保留和短语相关的视觉信息。这个方法在多个 MMT 数据集上实验结果表明显著优于强基线模型，尤其是在文本语境有限的情况下。