Recent research in the field of multimodal machine translation (MMT) has
indicated that the visual modality is either dispensable or offers only
marginal advantages. However, most of these conclusions are drawn from the
analysis of experimental results based on a limited set of bilingual
sentence-image pairs, such as Multi30k. In these kinds of datasets, the content
of one bilingual parallel sentence pair must be well represented by a manually
annotated image, which is different from the real-world translation scenario.
In this work, we adhere to the universal multimodal machine translation
framework proposed by Tang et al. (2022). This approach allows us to delve into
the impact of the visual modality on translation efficacy by leveraging
real-world translation datasets. Through a comprehensive exploration via
probing tasks, we find that the visual modality proves advantageous for the
majority of authentic translation datasets. Notably, the translation
performance primarily hinges on the alignment and coherence between textual and
visual contents. Furthermore, our results suggest that visual information
serves a supplementary role in multimodal translation and can be substituted.

近期在多模式机器翻译（MMT）领域的研究表明，视觉模态在翻译效果方面要么可有可无，要么只提供了边缘化优势。然而，这些结论大多来自于对有限的双语句子 - 图像配对数据集（如 Multi30k）的实验结果分析，而这类数据集中，一个双语平行句对的内容必须由手工标注的图像很好地表达，这与真实的翻译场景不同。在这项工作中，我们遵循了唐等人（2022 年）提出的通用多模式机器翻译框架。这种方法使我们能够通过利用真实世界的翻译数据集，深入研究视觉模态对翻译效果的影响。通过全面的探索性任务，我们发现视觉模态对大多数真实翻译数据集具有优势。值得注意的是，翻译性能主要取决于文本和视觉内容之间的对齐和一致性。此外，我们的结果表明，视觉信息在多模式翻译中发挥了补充作用，可以被替代。