Simultaneous machine translation (SiMT) aims to translate a continuous input
text stream into another language with the lowest latency and highest quality
possible. The translation thus has to start with an incomplete source text,
which is read progressively, creating the need for anticipation. In this paper,
we seek to understand whether the addition of visual information can compensate
for the missing source context. To this end, we analyse the impact of different
multimodal approaches and visual features on state-of-the-art SiMT frameworks.
Our results show that visual context is helpful and that visually-grounded
models based on explicit object region information are much better than
commonly used global features, reaching up to 3 BLEU points improvement under
low latency scenarios. Our qualitative analysis illustrates cases where only
the multimodal systems are able to translate correctly from English into
gender-marked languages, as well as deal with differences in word order, such
as adjective-noun placement between English and French.

本文旨在研究如何利用视觉信息来弥补缺失的源文本背景，评估不同的多模态方法和视觉特征对最先进的同时机器翻译 (SiMT) 框架的影响。结果表明，视觉背景是有帮助的，基于明确的物体区域信息的可视化模型优于常用的全局特征，在低延迟情境下的表现可提高 3 个 BLEU 分数。我们的定性分析展示，只有多模态系统才能正常翻译英语到标记性别的语言，并处理英语和法语之间的形容词和名词词序等差异。