Today, transformer language models serve as a core component for majority of
natural language processing tasks. Industrial application of such models
requires minimization of computation time and memory footprint. Knowledge
distillation is one of approaches to address this goal. Existing methods in
this field are mainly focused on reducing the number of layers or dimension of
embeddings/hidden representations. Alternative option is to reduce the number
of tokens in vocabulary and therefore the embeddings matrix of the student
model. The main problem with vocabulary minimization is mismatch between input
sequences and output class distributions of a teacher and a student models. As
a result, it is impossible to directly apply KL-based knowledge distillation.
We propose two simple yet effective alignment techniques to make knowledge
distillation to the students with reduced vocabulary. Evaluation of distilled
models on a number of common benchmarks for Russian such as Russian SuperGLUE,
SberQuAD, RuSentiment, ParaPhaser, Collection-3 demonstrated that our
techniques allow to achieve compression from $17\times$ to $49\times$, while
maintaining quality of $1.7\times$ compressed student with the full-sized
vocabulary, but reduced number of Transformer layers only. We make our code and
distilled models available.

该研究论文介绍了知识蒸馏的不同方法，其中包括对词汇表的缩减，以助于通过两种简单而有效的对齐技术，对具有简化词汇表的学生进行知识蒸馏，并证明这些技术可以在保持一定质量的情况下实现 17 倍至 49 倍的压缩。