Multilingual self-supervised speech representation models have greatly
enhanced the speech recognition performance for low-resource languages, and the
compression of these huge models has also become a crucial prerequisite for
their industrial application. In this paper, we propose DistilXLSR, a distilled
cross-lingual speech representation model. By randomly shuffling the phonemes
of existing speech, we reduce the linguistic information and distill
cross-lingual models using only English data. We also design a layer-jumping
initialization method to fully leverage the teacher's pre-trained weights.
Experiments on 2 kinds of teacher models and 15 low-resource languages show
that our method can reduce the parameters by 50% while maintaining
cross-lingual representation ability. Our method is proven to be generalizable
to various languages/teacher models and has the potential to improve the
cross-lingual performance of the English pre-trained models.

本文介绍了一种基于 DistilXLSR 的语音表征模型，通过随机打乱现有语音的音素，降低语言信息，在只使用英语数据的情况下，压缩跨语言模型并设计一种层级初始化方法，成功减少 50% 参数并在 15 种低资源语言和 2 种教师模型的实验中保持了跨语言表征能力，证明了其在各种语言 / 教师模型中具有普适性，有潜力提高英语预训练模型的跨语言性能。

DistilXLSR: 轻量级跨语言语音表示模型

DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model

Recent work has shown that small distilled language models are strong
competitors to models that are orders of magnitude larger and slower in a wide
range of information retrieval tasks. This has made distilled and dense models,
due to latency constraints, the go-to choice for deployment in real-world
retrieval applications. In this work, we question this practice by showing that
the number of parameters and early query-document interaction play a
significant role in the generalization ability of retrieval models. Our
experiments show that increasing model size results in marginal gains on
in-domain test sets, but much larger gains in new domains never seen during
fine-tuning. Furthermore, we show that rerankers largely outperform dense ones
of similar size in several tasks. Our largest reranker reaches the state of the
art in 12 of the 18 datasets of the Benchmark-IR (BEIR) and surpasses the
previous state of the art by 3 average points. Finally, we confirm that
in-domain effectiveness is not a good indicator of zero-shot effectiveness.
Code is available at
this https URL

本研究表明，在信息检索任务中，经过裁剪的小型语言模型是大型、速度慢得多的模型的强有力竞争者。在信息检索实际应用中，由于延迟限制，压缩和稠密模型成为首选。然而，通过实验，我们发现模型大小和早期查询文档交互对检索模型的泛化能力起着重要作用。增加模型大小在相同领域的测试数据集上几乎没有增益，但在从未在训练中见过的新领域上存在更大的提高。此外，我们还显示 reranker 在几个任务中大大优于其大小相似的稠密模型。我们的最大 reranker 在 Benchmark-IR（BEIR）的 18 个数据集中的 12 个数据集中达到了最先进水平，平均超过了以前的最优结果 3 个点。最后，我们证实，域内有效性不是零 - shot 有效性的好指标。