Multimodal machine translation (MMT) is a challenging task that seeks to
improve translation quality by incorporating visual information. However,
recent studies have indicated that the visual information provided by existing
MMT datasets is insufficient, causing models to disregard it and overestimate
their capabilities. This issue presents a significant obstacle to the
development of MMT research. This paper presents a novel solution to this issue
by introducing 3AM, an ambiguity-aware MMT dataset comprising 26,000 parallel
sentence pairs in English and Chinese, each with corresponding images. Our
dataset is specifically designed to include more ambiguity and a greater
variety of both captions and images than other MMT datasets. We utilize a word
sense disambiguation model to select ambiguous data from vision-and-language
datasets, resulting in a more challenging dataset. We further benchmark several
state-of-the-art MMT models on our proposed dataset. Experimental results show
that MMT models trained on our dataset exhibit a greater ability to exploit
visual information than those trained on other MMT datasets. Our work provides
a valuable resource for researchers in the field of multimodal learning and
encourages further exploration in this area. The data, code and scripts are
freely available at this https URL

通过引入 3AM 数据集，本文提出了一种新的解决方案来改善多模态机器翻译中存在的视觉信息不足的问题，并通过提供更具含糊性和更多种类的数据集，实现了对现有多模态机器翻译数据集更好的训练效果，进一步促进了多模态学习领域的研究和探索。

3AM: 一个模糊感知的多模式机器翻译数据集

3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset

With multilingual machine translation (MMT) models continuing to grow in size
and number of supported languages, it is natural to reuse and upgrade existing
models to save computation as data becomes available in more languages.
However, adding new languages requires updating the vocabulary, which
complicates the reuse of embeddings. The question of how to reuse existing
models while also making architectural changes to provide capacity for both old
and new languages has also not been closely studied. In this work, we introduce
three techniques that help speed up effective learning of the new languages and
alleviate catastrophic forgetting despite vocabulary and architecture
mismatches. Our results show that by (1) carefully initializing the network,
(2) applying learning rate scaling, and (3) performing data up-sampling, it is
possible to exceed the performance of a same-sized baseline model with 30%
computation and recover the performance of a larger model trained from scratch
with over 50% reduction in computation. Furthermore, our analysis reveals that
the introduced techniques help learn the new directions more effectively and
alleviate catastrophic forgetting at the same time. We hope our work will guide
research into more efficient approaches to growing languages for these MMT
models and ultimately maximize the reuse of existing models.

本论文提出了三种技术，可以加速新语言的学习并减轻灾难性遗忘，来提高多语言机器翻译模型的效率并最大化旧模型的重用，具体来说，这些技术包括仔细初始化网络、应用学习速率缩放、进行数据上采样。