Scaling pre-trained language models has resulted in large performance gains
in various natural language processing tasks but comes with a large cost in
memory requirements. Inspired by the position embeddings in transformers, we
aim to simplify and reduce the memory footprint of the multi-head attention
(MHA) mechanism. We propose an alternative module that uses only a single
shared projection matrix and multiple head embeddings (MHE), i.e. one per head.
We empirically demonstrate that our MHE attention is substantially more memory
efficient compared to alternative attention mechanisms while achieving high
predictive performance retention ratio to vanilla MHA on several downstream
tasks. MHE attention only requires a negligible fraction of additional
parameters ($3nd$, where $n$ is the number of attention heads and $d$ the size
of the head embeddings) compared to a single-head attention, while MHA requires
$(3n^2-3n)d^2-3nd$ additional parameters.

我们提出了一种使用单个共享投影矩阵和多个头嵌入（MHE）的替代模块，实验证明我们的 MHE 关注机制在多个下游任务上显著提高了内存效率，并且相对于 MHA 仅需要可忽略的附加参数。