We propose a simple modification to the conventional attention mechanism
applied by Transformers: Instead of quantifying pairwise query-key similarity
with scaled dot-products, we quantify it with the logarithms of scaled
dot-products of exponentials. Attention becomes expressible as a composition of
log-sums of exponentials that is linearizable, with a latent space of constant
size, enabling sequential application with constant time and space complexity
per token. We implement our modification, verify that it works in practice, and
conclude that it is a promising alternative to conventional attention.

我们对传统的 Transformer 模型中的注意力机制进行了简单修改，通过对指数函数的缩放点积取对数来量化查询 - 键的相似性，将注意力表达为一系列对数求和的指数函数，该方法的优势在于其线性化性质，具有恒定大小的潜在空间，每个标记的顺序应用具有恒定的时间和空间复杂度。我们实现了该修改，验证其在实践中的有效性，并得出结论，即它是传统注意力的一个有前途的替代方法。

具有恒定标记成本的 Softmax 注意力

Softmax Attention with Constant Cost per Token

We propose an alternative to the Turing test that removes the inherent
asymmetry between humans and machines in Turing's original imitation game. In
this new test, both humans and machines judge each other. We argue that this
makes the test more robust against simple deceptions. We also propose a small
number of refinements to improve further the test. These refinements could be
applied also to Turing's original imitation game.

我们提出了一种替代图灵测试的方法，通过使双方的人类和机器都参与评判从而消除原始模仿游戏中人与机器之间的固有不对称性，并且建议了一些改进来提高这种测试的鲁棒性。