While most current work in multimodal machine translation (MMT) uses the
Multi30k dataset for training and evaluation, we find that the resulting models
overfit to the Multi30k dataset to an extreme degree. Consequently, these
models perform very badly when evaluated against typical text-only testing sets
such as the WMT newstest datasets. In order to perform well on both Multi30k
and typical text-only datasets, we use a performant text-only machine
translation (MT) model as the starting point of our MMT model. We add
vision-text adapter layers connected via gating mechanisms to the MT model, and
incrementally transform the MT model into an MMT model by 1) pre-training using
vision-based masking of the source text and 2) fine-tuning on Multi30k.

对于目前的多模式机器翻译 (MMT) 工作来说，使用 Multi30k 数据集进行训练和评估导致模型过拟合到极高程度，从而在针对典型的纯文本测试集（如 WMT newstest 数据集）时表现非常差。为了在 Multi30k 和纯文本数据集上都表现出色，本文基于高效纯文本机器翻译 (MT) 模型作为 MMT 模型的起点，并通过使用视觉 - 文本适配器层和门控机制将 MT 模型逐步转换为 MMT 模型，通过 1) 使用基于视觉的源文本遮蔽进行预训练和 2) 在 Multi30k 上进行微调。