The softmax attention mechanism has emerged as a noteworthy development in
the field of Artificial Intelligence research, building on the successes of
Transformer-based architectures. However, their ever increasing sizes
necessitate ever increasing computational memory, that limits their usage. We
propose KgV, a sigmoid gating mechanism that, in conjunction with softmax
attention, significantly boosts performance without increasing architecture
size. To amend the size requirements, we leverage Tensor Chains to identify and
prune the excess parameters. We find that such excess resides primarily within
the embedding layer, and not in the output linear layer. To further improve
embedding and significantly reduce parameters, we introduce H-SoftPOS, a
hierarchical embedding layer which simultaneously enhances performance.
Remarkably, on the WMT14 English-German validation set, our approach yields a
threefold reduction in perplexity, surpassing the current state-of-the-art,
while reducing parameter counts also by a factor of 3. When we further reduce
the number of parameters up to sevenfold, we can still achieve a 21\% decrease
in perplexity with respect to the baseline Transformer. To understand
generalization capabilities, we conduct experiments on the 7 language pairs of
the WMT17 dataset. Our method outperforms existing techniques in terms of test
loss while simultaneously halving the number of parameters. Moreover, we
observe a 70 times reduction in variance with respect to the prior
state-of-the-art. In conclusion, our proposed method yields significant
improvements in performance and much lower memory cost. We call the resulting
architecture Anthe.

研究使用 KgV 提出了一种结合了 sigmoid 门控机制和 softmax 注意力机制的模型框架，同时采用张量链来减少嵌入层的超参数，并引入了 H-SoftPOS 当作嵌入层，相较于现有技术在性能和内存成本方面都有显著提高，并命名该体系结构为 Anthe。