With the growth of computing power neural machine translation (NMT) models also grow accordingly and become better. However, they also become harder to deploy on edge devices due to memory constraints. To cope with this problem, a common practice is to distill knowledge from a large and accurately-trained teacher network (T) into a compact student network (S). Although knowledge distillation (KD) is useful in most cases, our study shows that existing KD techniques might not be suitable enough for deep NMT engines, so we propose a novel alternative. In our model, besides matching T and S predictions we have a combinatorial mechanism to inject layer-level supervision from T to S. In this paper, we target low-resource settings and evaluate our translation engines for Portuguese--English, Turkish--English, and English--German directions. Students trained using our technique have 50% fewer parameters and can still deliver comparable results to those of 12-layer teachers.

本研究提出一种新颖的神经机器翻译知识蒸馏技术，通过层级监督，将大型准确的老师网络的知识注入到小巧的学生网络中，以适应边缘设备上内存约束的低资源环境。实验表明，采用该技术，学生网络的参数数量减少了50％，仍能提供与老师网络相媲美的翻译结果。针对葡萄牙语-英语、土耳其语-英语和英语-德语方向的翻译实验结果也得到了验证。

为何跳过合并：一种用于中间层的简单知识蒸馏技术