There has been a growing interest in developing multimodal machine
translation (MMT) systems that enhance neural machine translation (NMT) with
visual knowledge. This problem setup involves using images as auxiliary
information during training, and more recently, eliminating their use during
inference. Towards this end, previous works face a challenge in training
powerful MMT models from scratch due to the scarcity of annotated multilingual
vision-language data, especially for low-resource languages. Simultaneously,
there has been an influx of multilingual pre-trained models for NMT and
multimodal pre-trained models for vision-language tasks, primarily in English,
which have shown exceptional generalisation ability. However, these are not
directly applicable to MMT since they do not provide aligned multimodal
multilingual features for generative tasks. To alleviate this issue, instead of
designing complex modules for MMT, we propose CLIPTrans, which simply adapts
the independently pre-trained multimodal M-CLIP and the multilingual mBART. In
order to align their embedding spaces, mBART is conditioned on the M-CLIP
features by a prefix sequence generated through a lightweight mapping network.
We train this in a two-stage pipeline which warms up the model with image
captioning before the actual translation task. Through experiments, we
demonstrate the merits of this framework and consequently push forward the
state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The
code can be found at www.github.com/devaansh100/CLIPTrans.

本研究提出了 CLIPTrans 框架，通过简单地调整预训练的多模态 M-CLIP 和多语言 mBART 模型，使它们的嵌入空间对齐，并通过轻量级映射网络对 mBART 进行条件化。实验证明该框架的优点，并将标准基准提升了平均 2.67 个 BLEU 值。