Previous studies have proved that cross-lingual knowledge distillation can
significantly improve the performance of pre-trained models for cross-lingual
similarity matching tasks. However, the student model needs to be large in this
operation. Otherwise, its performance will drop sharply, thus making it
impractical to be deployed to memory-limited devices. To address this issue, we
delve into cross-lingual knowledge distillation and propose a multi-stage
distillation framework for constructing a small-size but high-performance
cross-lingual model. In our framework, contrastive learning, bottleneck, and
parameter recurrent strategies are combined to prevent performance from being
compromised during the compression process. The experimental results
demonstrate that our method can compress the size of XLM-R and MiniLM by more
than 50\%, while the performance is only reduced by about 1%.

本论文提出了一种多阶段蒸馏框架来构建小型但高性能的跨语言模型，同时结合对比学习、瓶颈和参数循环策略，解决了模型压缩过程中性能损失严重的问题，并证明该方法可以将 XLM-R 和 MiniLM 的大小压缩了 50％以上，而性能仅降低了约 1％。