Building ASR models across many languages is a challenging multi-task
learning problem due to large variations and heavily unbalanced data. Existing
work has shown positive transfer from high resource to low resource languages.
However, degradations on high resource languages are commonly observed due to
interference from the heterogeneous multilingual data and reduction in
per-language capacity. We conduct a capacity study on a 15-language task, with
the amount of data per language varying from 7.6K to 53.5K hours. We adopt
GShard [1] to efficiently scale up to 10B parameters. Empirically, we find that
(1) scaling the number of model parameters is an effective way to solve the
capacity bottleneck - our 500M-param model already outperforms monolingual
baselines and scaling it to 1B and 10B brought further quality gains; (2)
larger models are not only more data efficient, but also more efficient in
terms of training cost as measured in TPU days - the 1B-param model reaches the
same accuracy at 34% of training time as the 500M-param model; (3) given a
fixed capacity budget, adding depth works better than width and large encoders
do better than large decoders; (4) with continuous training, they can be
adapted to new languages and domains.

本篇研究旨在研究 15 种任务下，训练语言模型的容量瓶颈，并发现通过增加模型参数，添加深度，以及使用大型编码器等方法可以增加训练效率。

针对大规模多语言 ASR 的端到端模型扩展

Scaling End-to-End Models for Large-Scale Multilingual ASR

Neural network scaling has been critical for improving the model quality in
many real-world machine learning applications with vast amounts of training
data and compute. Although this trend of scaling is affirmed to be a sure-fire
approach for better model quality, there are challenges on the path such as the
computation cost, ease of programming, and efficient implementation on parallel
devices. GShard is a module composed of a set of lightweight annotation APIs
and an extension to the XLA compiler. It provides an elegant way to express a
wide range of parallel computation patterns with minimal changes to the
existing model code. GShard enabled us to scale up multilingual neural machine
translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600
billion parameters using automatic sharding. We demonstrate that such a giant
model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to
achieve far superior quality for translation from 100 languages to English
compared to the prior art.

利用 GShard 能够高效地扩展神经网络规模，支持多语言机器翻译，使得能够在 4 天内用 TPU v3 加速器训练 600 亿参数的模型，并且在从 100 种语言翻译成英语方面的性能要明显优于之前的工作。