Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g.,
redundancy and over-parameterization. Specifically, the heads of MHA were
originally designed to attend to information from different representation
subspaces, whereas prior studies found that some attention heads likely learn
similar features and can be pruned without harming performance. Inspired by the
minimum-redundancy feature selection, we assume that focusing on the most
representative and distinctive features with minimum resources can mitigate the
above issues and lead to more effective and efficient MHAs. In particular, we
propose Grouped Head Attention, trained with a self-supervised group constraint
that group attention heads, where each group focuses on an essential but
distinctive feature subset. We additionally propose a Voting-to-Stay procedure
to remove redundant heads, thus achieving a transformer with lighter weights.
Moreover, our method achieves significant performance gains on three
well-established tasks while considerably compressing parameters.

本文提出了使用最小冗余特征选择的思想，通过聚焦于最具代表性和独特性的特征，采用自监督群组约束进行训练的 Grouped Head Attention 模型，以及通过投票去除冗余头部来实现更有效和高效的多头自注意力模型。实验结果表明，该方法在三个基准任务上显著提高性能的同时，压缩参数。