In this paper, we introduce the Hyperbolic Tangent Exponential Linear Unit
(TeLU), a novel neural network activation function, represented as $f(x) =
x{\cdot}tanh(e^x)$. TeLU is designed to overcome the limitations of
conventional activation functions like ReLU, GELU, and Mish by addressing the
vanishing and, to an extent, the exploding gradient problems. Our theoretical
analysis and empirical assessments reveal that TeLU outperforms existing
activation functions in stability and robustness, effectively adjusting
activation outputs' mean towards zero for enhanced training stability and
convergence. Extensive evaluations against popular activation functions (ReLU,
GELU, SiLU, Mish, Logish, Smish) across advanced architectures, including
Resnet-50, demonstrate TeLU's lower variance and superior performance, even
under hyperparameter conditions optimized for other functions. In large-scale
tests with challenging datasets like CIFAR-10, CIFAR-100, and TinyImageNet,
encompassing 860 scenarios, TeLU consistently showcased its effectiveness,
positioning itself as a potential new standard for neural network activation
functions, boosting stability and performance in diverse deep learning
applications.

引入了 Hyperbolic Tangent Exponential Linear Unit（TeLU）作为一种新的神经网络激活函数，通过解决渐变消失和渐变爆炸问题，提高了稳定性和鲁棒性，在各种深度学习应用中表现出卓越的性能，将其视为潜在的新标准。

稳定且鲁棒的深度学习方法：双曲正切指数线性单元（TeLU）

Stable and Robust Deep Learning By Hyperbolic Tangent Exponential Linear  Unit (TeLU)

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural
network activation function. The GELU activation function is $x\Phi(x)$, where
$\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU
nonlinearity weights inputs by their value, rather than gates inputs by their
sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of
the GELU nonlinearity against the ReLU and ELU activations and find performance
improvements across all considered computer vision, natural language
processing, and speech tasks.

本文提出了高性能的神经网络激活函数 ——Gaussian Error Linear Unit（GELU），它的非线性性能优于 ReLU 和 ELU，并在所有涉及的计算机视觉、自然语言处理和语音任务中均实现了性能提升。