This paper presents an in-depth study of multimodal machine translation
(MMT), examining the prevailing understanding that MMT systems exhibit
decreased sensitivity to visual information when text inputs are complete.
Instead, we attribute this phenomenon to insufficient cross-modal interaction,
rather than image information redundancy. A novel approach is proposed to
generate parallel Visual Question-Answering (VQA) style pairs from the source
text, fostering more robust cross-modal interaction. Using Large Language
Models (LLMs), we explicitly model the probing signal in MMT to convert it into
VQA-style data to create the Multi30K-VQA dataset. An MMT-VQA multitask
learning framework is introduced to incorporate explicit probing signals from
the dataset into the MMT training process. Experimental results on two
widely-used benchmarks demonstrate the effectiveness of this novel approach.
Our code and data would be available at:
https://github.com/libeineu/MMT-VQA.

本研究论文通过深入研究多模式机器翻译（MMT），探讨了 MMT 系统在源文本完整时对视觉信息的敏感性降低的现象，并提出了一种新的方法来生成并行的视觉问答（VQA）风格对，以促进更强大的跨模态交互。使用大型语言模型（LLMs），将 MMT 中的探测信号显式建模为 VQA 风格数据，创建了 Multi30K-VQA 数据集，并引入了 MMT-VQA 多任务学习框架，将来自数据集的显式探测信号纳入 MMT 训练过程。在两个广泛使用的基准测试中验证了该新方法的有效性。本文提供的代码和数据可在 https://github.com/libeineu/MMT-VQA 获取。