There are common semantics shared across text and images. Given a sentence in
a source language, whether depicting the visual scene helps translation into a
target language? Existing multimodal neural machine translation methods (MNMT)
require triplets of bilingual sentence - image for training and tuples of
source sentence - image for inference. In this paper, we propose ImagiT, a
novel machine translation method via visual imagination. ImagiT first learns to
generate visual representation from the source sentence, and then utilizes both
source sentence and the "imagined representation" to produce a target
translation. Unlike previous methods, it only needs the source sentence at the
inference time. Experiments demonstrate that ImagiT benefits from visual
imagination and significantly outperforms the text-only neural machine
translation baselines. Further analysis reveals that the imagination process in
ImagiT helps fill in missing information when performing the degradation
strategy.

本文提出一种基于视觉想象的神经机器翻译方法 ImagiT，仅需要源语言句子即可生成目标翻译，并证明该方法较传统文本翻译方法表现更好，揭示想象过程在修复缺失信息时的重要性。

生成性想象提高机器翻译

Generative Imagination Elevates Machine Translation

Simultaneous translation involves translating a sentence before the speaker's
utterance is completed in order to realize real-time understanding in multiple
languages. This task is significantly more challenging than the general full
sentence translation because of the shortage of input information during
decoding. To alleviate this shortage, we propose multimodal simultaneous neural
machine translation (MSNMT), which leverages visual information as an
additional modality. Our experiments with the Multi30k dataset showed that
MSNMT significantly outperforms its text-only counterpart in more timely
translation situations with low latency. Furthermore, we verified the
importance of visual information during decoding by performing an adversarial
evaluation of MSNMT, where we studied how models behaved with incongruent input
modality and analyzed the effect of different word order between source and
target languages.

提出了一种多模式神经机器翻译方法，即多模式同时翻译 (MSNMT)，能够在翻译过程中利用视频信息和实现实时多语言理解。实验证实，该方法可显著提高翻译效率，特别是在低延迟情况下，比单纯文本的模型更加优秀。此外，我们通过对 MSNMT 的对抗性评估研究发现了视频信息在解码过程中的重要性，并分析了源语言和目标语言之间不同的单词顺序对翻译结果的影响。

面向多模态同时神经机器翻译

Towards Multimodal Simultaneous Neural Machine Translation

In recent years, pretrained word embeddings have proved useful for multimodal
neural machine translation (NMT) models to address the shortage of available
datasets. However, the integration of pretrained word embeddings has not yet
been explored extensively. Further, pretrained word embeddings in high
dimensional spaces have been reported to suffer from the hubness problem.
Although some debiasing techniques have been proposed to address this problem
for other natural language processing tasks, they have seldom been studied for
multimodal NMT models. In this study, we examine various kinds of word
embeddings and introduce two debiasing techniques for three multimodal NMT
models and two language pairs -- English-German translation and English-French
translation. With our optimal settings, the overall performance of multimodal
models was improved by up to +1.93 BLEU and +2.02 METEOR for English-German
translation and +1.73 BLEU and +0.95 METEOR for English-French translation.

本研究探讨了预训练词嵌入及其在多模式神经机器翻译模型中的应用，引入两种去偏差技术，并在英德翻译和英法翻译两种语言对上获得了最多 + 1.93 BLEU 和 + 2.02 METEOR 的性能提升。

消除词嵌入中的偏见改善多模式机器翻译

Debiasing Word Embeddings Improves Multimodal Machine Translation

In Multimodal Neural Machine Translation (MNMT), a neural model generates a
translated sentence that describes an image, given the image itself and one
source descriptions in English. This is considered as the multimodal image
caption translation task. The images are processed with Convolutional Neural
Network (CNN) to extract visual features exploitable by the translation model.
So far, the CNNs used are pre-trained on object detection and localization
task. We hypothesize that richer architecture, such as dense captioning models,
may be more suitable for MNMT and could lead to improved translations. We
extend this intuition to the word-embeddings, where we compute both linguistic
and visual representation for our corpus vocabulary. We combine and compare
different confi

本文探讨了在多模态神经机器翻译 (MNMT) 中使用密集标注模型进行视觉特征提取和词嵌入，以提高图像描述翻译模型的效果。