In Multimodal Neural Machine Translation (MNMT), a neural model generates a
translated sentence that describes an image, given the image itself and one
source descriptions in English. This is considered as the multimodal image
caption translation task. The images are processed with Convolutional Neural
Network (CNN) to extract visual features exploitable by the translation model.
So far, the CNNs used are pre-trained on object detection and localization
task. We hypothesize that richer architecture, such as dense captioning models,
may be more suitable for MNMT and could lead to improved translations. We
extend this intuition to the word-embeddings, where we compute both linguistic
and visual representation for our corpus vocabulary. We combine and compare
different confi

本文探讨了在多模态神经机器翻译 (MNMT) 中使用密集标注模型进行视觉特征提取和词嵌入，以提高图像描述翻译模型的效果。

通过丰富视觉特征和视觉驱动的词向量提升多模态神经机器翻译

Visually Grounded Word Embeddings and Richer Visual Features for  Improving Multimodal Neural Machine Translation

In state-of-the-art Neural Machine Translation, an attention mechanism is
used during decoding to enhance the translation. At every step, the decoder
uses this mechanism to focus on different parts of the source sentence to
gather the most useful information before outputting its target word. Recently,
the effectiveness of the attention mechanism has also been explored for
multimodal tasks, where it becomes possible to focus both on sentence parts and
image regions. Approaches to pool two modalities usually include element-wise
product, sum or concatenation. In this paper, we evaluate the more advanced
Multimodal Compact Bilinear pooling method, which takes the outer product of
two vectors to combine the attention features for the two modalities. This has
been previously investigated for visual question answering. We try out this
approach for multimodal image caption translation and show improvements
compared to basic combination methods.

本文探讨了在多模态翻译中使用复合双线性池化方法的效果，通过将两种注意力特征进行外积组合，相比于基本的组合方法，其对于图像字幕翻译的表现有所提升。