We propose gated language experts to improve multilingual transformer
transducer models without any language identification (LID) input from users
during inference. We define gating mechanism and LID loss to let transformer
encoders learn language-dependent information, construct the multilingual
transformer block with gated transformer experts and shared transformer layers
for compact models, and apply linear experts on joint network output to better
regularize speech acoustic and token label joint information. Furthermore, a
curriculum training scheme is proposed to let LID guide the gated language
experts for better serving their corresponding languages. Evaluated on the
English and Spanish bilingual task, our methods achieve average 12.5% and 7.3%
relative word error reductions over the baseline bilingual model and
monolingual models, respectively, obtaining similar results to the upper bound
model trained and inferred with oracle LID. We further explore our method on
trilingual, quadrilingual, and pentalingual models, and observe similar
advantages as in the bilingual models, which demonstrates the easy extension to
more languages.

使用门控语言专家和课程训练方案改善多语种 Transformer 转导模型的性能，并在英语和西班牙语的双语任务中比基准双语和单语模型分别获得 12.5% 和 7.3% 的相对字错误率降低，并且扩展到更多语言也具有类似的优势。

使用门控语言专家和课程训练建立高精度的多语言 ASR

Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training

The objective of this paper is self-supervised learning of spatio-temporal
embeddings from video, suitable for human action recognition. We make three
contributions: First, we introduce the Dense Predictive Coding (DPC) framework
for self-supervised representation learning on videos. This learns a dense
encoding of spatio-temporal blocks by recurrently predicting future
representations; Second, we propose a curriculum training scheme to predict
further into the future with progressively less temporal context. This
encourages the model to only encode slowly varying spatial-temporal signals,
therefore leading to semantic representations; Third, we evaluate the approach
by first training the DPC model on the Kinetics-400 dataset with
self-supervised learning, and then finetuning the representation on a
downstream task, i.e. action recognition. With single stream (RGB only), DPC
pretrained representations achieve state-of-the-art self-supervised performance
on both UCF101(75.7% top1 acc) and HMDB51(35.7% top1 acc), outperforming all
previous learning methods by a significant margin, and approaching the
performance of a baseline pre-trained on ImageNet.

本论文介绍了自监督学习，通过视频中的 Dense Predictive Coding (DPC) 框架形成具有密集的时空块的编码，利用课程训练方式预测未来，并在 Kinetics-400 数据集上进行实验，结果表明，DPC 的自监督预训练性能超过了以前的方法，并接近于在 ImageNet 上预先训练的基线。