With multilingual machine translation (MMT) models continuing to grow in size
and number of supported languages, it is natural to reuse and upgrade existing
models to save computation as data becomes available in more languages.
However, adding new languages requires updating the vocabulary, which
complicates the reuse of embeddings. The question of how to reuse existing
models while also making architectural changes to provide capacity for both old
and new languages has also not been closely studied. In this work, we introduce
three techniques that help speed up effective learning of the new languages and
alleviate catastrophic forgetting despite vocabulary and architecture
mismatches. Our results show that by (1) carefully initializing the network,
(2) applying learning rate scaling, and (3) performing data up-sampling, it is
possible to exceed the performance of a same-sized baseline model with 30%
computation and recover the performance of a larger model trained from scratch
with over 50% reduction in computation. Furthermore, our analysis reveals that
the introduced techniques help learn the new directions more effectively and
alleviate catastrophic forgetting at the same time. We hope our work will guide
research into more efficient approaches to growing languages for these MMT
models and ultimately maximize the reuse of existing models.

本论文提出了三种技术，可以加速新语言的学习并减轻灾难性遗忘，来提高多语言机器翻译模型的效率并最大化旧模型的重用，具体来说，这些技术包括仔细初始化网络、应用学习速率缩放、进行数据上采样。

高效升级多语言机器翻译模型以支持更多语言

Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages

Theory of convolutional neural networks suggests the property of shift
equivariance, i.e., that a shifted input causes an equally shifted output. In
practice, however, this is not always the case. This poses a great problem for
scene text detection for which a consistent spatial response is crucial,
irrespective of the position of the text in the scene.
Using a simple synthetic experiment, we demonstrate the inherent shift
variance of a state-of-the-art fully convolutional text detector. Furthermore,
using the same experimental setting, we show how small architectural changes
can lead to an improved shift equivariance and less variation of the detector
output. We validate the synthetic results using a real-world training schedule
on the text detection network. To quantify the amount of shift variability, we
propose a metric based on well-established text detection benchmarks.
While the proposed architectural changes are not able to fully recover shift
equivariance, adding smoothing filters can substantially improve shift
consistency on common text datasets. Considering the potentially large impact
of small shifts, we propose to extend the commonly used text detection metrics
by the metric described in this work, in order to be able to quantify the
consistency of text detectors.

通过一个简单的合成实验，我们展示了最先进的全卷积文字检测器固有的移位差异性。我们展示了如何进行小的体系结构更改，可以带来改善的移位平稳性和检测器输出的变化较小。我们提出了一个基于深度学习已有的文字检测基准的度量，用于量化移位可变性的程度。尽管所提出的架构更改不能完全恢复移位等价性，但添加平滑滤波器可以大大改善常见的文本数据集上的移位一致性。考虑到小偏移可能具有巨大的影响，我们建议将此工作中描述的指标扩展到常用的文本检测度量中，以便能够量化文本检测器的一致性。