There is a growing amount of literature on the relationship between wide
neural networks (NNs) and Gaussian processes (GPs), identifying an equivalence
between the two for a variety of NN architectures. This equivalence enables,
for instance, accurate approximation of the behaviour of wide Bayesian NNs
without MCMC or variational approximations, or characterisation of the
distribution of randomly initialised wide NNs optimised by gradient descent
without ever running an optimiser. We provide a rigorous extension of these
results to NNs involving attention layers, showing that unlike single-head
attention, which induces non-Gaussian behaviour, multi-head attention
architectures behave as GPs as the number of heads tends to infinity. We
further discuss the effects of positional encodings and layer normalisation,
and propose modifications of the attention mechanism which lead to improved
results for both finite and infinitely wide NNs. We evaluate attention kernels
empirically, leading to a moderate improvement upon the previous
state-of-the-art on CIFAR-10 for GPs without trainable kernels and advanced
data preprocessing. Finally, we introduce new features to the Neural Tangents
library (Novak et al., 2020) allowing applications of NNGP/NTK models, with and
without attention, to variable-length sequences, with an example on the IMDb
reviews dataset.

通过 NNs 和 GPs 之间的等价性，研究多头关注层体系结构并评估其在 GP 中的应用，通过对注意力内核的评估，提出了改进的注意力机制，并介绍了新特性，将 NNGP / NTK 模型应用于变长序列。

无限关注：NNGP 和 NTK 用于深度注意力网络

Infinite attention: NNGP and NTK for deep attention networks

Named entity recognition (NER) systems that perform well require task-related
and manually annotated datasets. However, they are expensive to develop, and
are thus limited in size. As there already exists a large number of NER
datasets that share a certain degree of relationship but differ in content, it
is important to explore the question of whether such datasets can be combined
as a simple method for improving NER performance. To investigate this, we
developed a novel locally detecting multitask model using FFNNs. The model
relies on encoding variable-length sequences of words into theoretically
lossless and unique fixed-size representations. We applied this method to
several well-known NER tasks and compared the results of our model to baseline
models as well as other published results. As a result, we observed competitive
performance in nearly all of the tasks.

本研究尝试通过组合多个具有一定关系但不同内容的命名实体识别数据集，使用基于 FFNN 的局部探测多任务模型从变长单词序列生成理论上无损且唯一的固定大小表示，提高该领域任务的性能表现，实验结果表明，该模型在几乎所有任务中表现出有竞争力的性能。