The advent of scalable deep models and large datasets has improved the
performance of Neural Machine Translation. Knowledge Distillation (KD) enhances
efficiency by transferring knowledge from a teacher model to a more compact
student model. However, KD approaches to Transformer architecture often rely on
heuristics, particularly when deciding which teacher layers to distill from. In
this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to
address the feature mapping problem by adaptively aligning student attention
heads with their teacher counterparts during training. The Attention Alignment
Module in A2D performs a dense head-by-head comparison between student and
teacher attention heads across layers, turning the combinatorial mapping
heuristics into a learning problem. Our experiments show the efficacy of A2D,
demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb
and WMT-2014 En->De, respectively, compared to Transformer baselines.

通过学生模型与教师模型的自适应对齐，'Align-to-Distill'（A2D）策略在 Transformer 架构的知识蒸馏中解决了特征映射问题，实验证明 A2D 相较于 Transformer 基准模型，在 WMT-2022 的 De->Dsb 和 WMT-2014 的 En->De 翻译任务中分别获得了 + 3.61 和 + 0.63 BLEU 得分的提升。