Attention layers are widely used in natural language processing (NLP) and are
beginning to influence computer vision architectures. Training very large
transformer models allowed significant improvement in both fields, but once
trained, these networks show symptoms of over-parameterization. For instance,
it is known that many attention heads can be pruned without impacting accuracy.
This work aims to enhance current understanding on how multiple heads interact.
Motivated by the observation that attention heads learn redundant key/query
projections, we propose a collaborative multi-head attention layer that enables
heads to learn shared projections. Our scheme decreases the number of
parameters in an attention layer and can be used as a drop-in replacement in
any transformer architecture. Our experiments confirm that sharing key/query
dimensions can be exploited in language understanding, machine translation and
vision. We also show that it is possible to re-parametrize a pre-trained
multi-head attention layer into our collaborative attention layer.
Collaborative multi-head attention reduces the size of the key and query
projections by 4 for same accuracy and speed. Our code is public.

该论文提出了一种协作式多头注意力层，该方法通过共享 key/query 投影来降低注意力层中参数的数量，可以用于任何变压器体系结构，并对语言理解、机器翻译和计算机视觉方面进行了验证和实验，并可将预训练的多头注意力层重新参数化为协同注意力层，使 key 和 query 投影的大小减小了 4 倍，而准确性和速度保持不变。