BriefGPT.xyz
Feb, 2020
基于Transformer的机器翻译中固定编码器自注意力模式
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
HTML
PDF
Alessandro Raganato, Yves Scherrer, Jörg Tiedemann
TL;DR
使用简单固定的关注模式替换Transformer中编码器层的大多数注意力头对神经机器翻译的质量没有影响，并且在资源匮乏的情况下，甚至可以将BLEU评分提高3个点。
Abstract
transformer-based models
have brought a radical change to
neural machine translation
. A key feature of the Transformer architecture is the so-called
→