Unveiling the reasons behind the exceptional success of transformers requires
a better understanding of why attention layers are suitable for NLP tasks. In
particular, such tasks require predictive models to capture contextual meaning
which often depends on one or few words, even if the sentence is long. Our work
studies this key property, dubbed word sensitivity (WS), in the prototypical
setting of random features. We show that attention layers enjoy high WS,
namely, there exists a vector in the space of embeddings that largely perturbs
the random attention features map. The argument critically exploits the role of
the softmax in the attention layer, highlighting its benefit compared to other
activations (e.g., ReLU). In contrast, the WS of standard random features is of
order $1/\sqrt{n}$, $n$ being the number of words in the textual sample, and
thus it decays with the length of the context. We then translate these results
on the word sensitivity into generalization bounds: due to their low WS, random
features provably cannot learn to distinguish between two sentences that differ
only in a single word; in contrast, due to their high WS, random attention
features have higher generalization capabilities. We validate our theoretical
results with experimental evidence over the BERT-Base word embeddings of the
imdb review dataset.

通过对随机特征的原型设定研究词敏感性，我们展示了注意力层在自然语言处理任务中具有较高的词敏感性，同时更好地理解了为何注意力层适用于这些任务，从而揭示出它相比其他激活函数（如 ReLU）具有的优势。同时，我们将词敏感性的结果转化为泛化界限，验证了随机注意力特征具有较高泛化能力的理论结果，并在 imdb 评论数据集中对 BERT-Base 词嵌入进行实证验证。