Visual context provides grounding information for multimodal machine
translation (MMT). However, previous MMT models and probing studies on visual
features suggest that visual information is less explored in MMT as it is often
redundant to textual information. In this paper, we propose an object-level
visual context modeling framework (OVC) to efficiently capture and explore
visual information for multimodal machine translation. With detected objects,
the proposed OVC encourages MMT to ground translation on desirable visual
objects by masking irrelevant objects in the visual modality. We equip the
proposed with an additional object-masking loss to achieve this goal. The
object-masking loss is estimated according to the similarity between masked
objects and the source texts so as to encourage masking source-irrelevant
objects. Additionally, in order to generate vision-consistent target words, we
further propose a vision-weighted translation loss for OVC. Experiments on MMT
datasets demonstrate that the proposed OVC model outperforms state-of-the-art
MMT models and analyses show that masking irrelevant objects helps grounding in
MMT.

本文提出了一种基于物体级别视觉上下文建模的多模态机器翻译框架，采用对象掩模技术在可视模态下掩盖与源文本不相关的对象以实现翻译的视觉基础，同时引入了视觉加权翻译损失提高目标语言的视觉一致性，实验表明该模型优于现有的多模态机器翻译模型。