In this paper we provide, to the best of our knowledge, the first
comprehensive approach for incorporating various masking mechanisms into
Transformers architectures in a scalable way. We show that recent results on
linear causal attention (Choromanski et al., 2021) and log-linear RPE-attention
(Luo et al., 2021) are special cases of this general mechanism. However by
casting the problem as a topological (graph-based) modulation of unmasked
attention, we obtain several results unknown before, including efficient
d-dimensional RPE-masking and graph-kernel masking. We leverage many
mathematical techniques ranging from spectral analysis through dynamic
programming and random walks to new algorithms for solving Markov processes on
graphs. We provide a corresponding empirical evaluation.

本文提供了一种全面的方法，可以在可扩展的方式中将各种掩码机制纳入 Transformers 架构。我们提出了一种基于拓扑的建模方法，可以生成高效的 d 维 RPE 掩码和图核心掩码。