Pretrained multilingual large language models have typically used heuristic
temperature-based sampling to balance between different languages. However
previous work has not systematically evaluated the efficacy of different
pretraining language distributions across model scales. In this paper, we
propose a new sampling method, UniMax, that delivers more uniform coverage of
head languages while mitigating overfitting on tail languages by explicitly
capping the number of repeats over each language's corpus. We perform an
extensive series of ablations testing a range of sampling strategies on a suite
of multilingual benchmarks, while varying model scale. We find that UniMax
outperforms standard temperature-based sampling, and the benefits persist as
scale increases. As part of our contribution, we release: (i) an improved and
refreshed mC4 multilingual corpus consisting of 29 trillion characters across
107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained
with UniMax sampling.

本文提出了一种新的采样方法 UniMax，可在平衡语言之间的差异并减轻尾部语言的过拟合的同时，提供更加均匀的头部语言覆盖，并在一系列多语言评估基准测试中证明了 UniMax 的优越性和其随着模型规模的增加而持续的优点。

UniMax: 大规模多语言预训练中更公平、更有效的语言采样

UniMax: Fairer and more Effective Language Sampling for Large-Scale  Multilingual Pretraining

Code-switching automatic speech recognition becomes one of the most
challenging and the most valuable scenarios of automatic speech recognition,
due to the code-switching phenomenon between multilingual language and the
frequent occurrence of code-switching phenomenon in daily life. The ISCSLP 2022
Chinese-English Code-Switching Automatic Speech Recognition (CSASR) Challenge
aims to promote the development of code-switching automatic speech recognition.
The ISCSLP 2022 CSASR challenge provided two training sets, TAL_CSASR corpus
and MagicData-RAMC corpus, a development and a test set for participants, which
are used for CSASR model training and evaluation. Along with the challenge, we
also provide the baseline system performance for reference. As a result, more
than 40 teams participated in this challenge, and the winner team achieved
16.70% Mixture Error Rate (MER) performance on the test set and has achieved
9.8% MER absolute improvement compared with the baseline system. In this paper,
we will describe the datasets, the associated baselines system and the
requirements, and summarize the CSASR challenge results and major techniques
and tricks used in the submitted systems.

本文介绍了 ISCSLP 2022 中英文混合自动语音识别（CSASR）挑战赛，包括训练集、开发集和测试集等数据，以及参赛队伍的表现和技术。其中获胜队伍在测试集上取得了 16.70% 的混合误差率（MER）表现，并且相对基准系统取得了 9.8% 的 MER 绝对改善。