Activation functions are core components of all deep learning architectures.
Currently, the most popular activation functions are smooth ReLU variants like
GELU and SiLU. These are self-gated activation functions where the range of the
gating function is between zero and one. In this paper, we explore the
viability of using arctan as a gating mechanism. A self-gated activation
function that uses arctan as its gating function has a monotonically increasing
first derivative. To make this activation function competitive, it is necessary
to introduce a trainable parameter for every MLP block to expand the range of
the gating function beyond zero and one. We find that this technique also
improves existing self-gated activation functions. We conduct an empirical
evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and
Expanded SiLU (xSiLU) and show that they outperform existing activation
functions within a transformer architecture. Additionally, expanded gating
ranges show promising results in improving first-order Gated Linear Units
(GLU).

探索使用反正切作为门控机制的自门控激活函数，该函数具有单调递增的一阶导数，通过引入每个 MLP 块的可训练参数来扩展门控函数的范围，实验证明这种技术改善了现有的自门控激活函数，并在 Transformer 架构中显示出优越性能。扩展的门控范围还显示了改善一阶门控线性单元 (GLU) 的有希望的结果。