While impressive performance has been achieved on the task of Answer Sentence
Selection (AS2) for English, the same does not hold for languages that lack
large labeled datasets. In this work, we propose Cross-Lingual Knowledge
Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2
models for low-resource languages in the tasks without the need of labeled data
for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a
translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a
multilingual AS2 dataset with over 70K questions spanning 8 typologically
diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2
with multiple teachers, diverse monolingual and multilingual pretrained
language models (PLMs) as students, and both monolingual and multilingual
training. The results demonstrate that CLKD either outperforms or rivals even
supervised fine-tuning with the same amount of labeled data and a combination
of machine translation and the teacher model. Our method can potentially enable
stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the
largest multilingual AS2 dataset for further studies in the research community.

本文提出了一种名为 Cross-Lingual Knowledge Distillation (CLKD) 的方法，使用强大的英文 Answer Sentence Selection（AS2）模型作为老师进行跨语言知识蒸馏，可用于训练对于标注数据有限的低资源语言的 AS2 模型。作者还介绍了两个多语言数据集进行了实证研究，通过多次实验验证，CLKD 方法在不用标注数据的情况下，可与使用相同数量标注数据的监督微调方法相媲美，具有潜力为低资源语言提供更强大的 AS2 模型，同时这两个多语言数据集能够在 AS2 领域进行更广泛的研究。

低资源语言下跨语言知识蒸馏的答案句子选择

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in  Low-Resource Languages

Previous studies have proved that cross-lingual knowledge distillation can
significantly improve the performance of pre-trained models for cross-lingual
similarity matching tasks. However, the student model needs to be large in this
operation. Otherwise, its performance will drop sharply, thus making it
impractical to be deployed to memory-limited devices. To address this issue, we
delve into cross-lingual knowledge distillation and propose a multi-stage
distillation framework for constructing a small-size but high-performance
cross-lingual model. In our framework, contrastive learning, bottleneck, and
parameter recurrent strategies are combined to prevent performance from being
compromised during the compression process. The experimental results
demonstrate that our method can compress the size of XLM-R and MiniLM by more
than 50\%, while the performance is only reduced by about 1%.

本论文提出了一种多阶段蒸馏框架来构建小型但高性能的跨语言模型，同时结合对比学习、瓶颈和参数循环策略，解决了模型压缩过程中性能损失严重的问题，并证明该方法可以将 XLM-R 和 MiniLM 的大小压缩了 50％以上，而性能仅降低了约 1％。