The success of artificial neural networks (ANNs) hinges greatly on the
judicious selection of an activation function, introducing non-linearity into
network and enabling them to model sophisticated relationships in data.
However, the search of activation functions has largely relied on empirical
knowledge in the past, lacking theoretical guidance, which has hindered the
identification of more effective activation functions. In this work, we offer a
proper solution to such issue. Firstly, we theoretically demonstrate the
existence of the worst activation function with boundary conditions (WAFBC)
from the perspective of information entropy. Furthermore, inspired by the
Taylor expansion form of information entropy functional, we propose the
Entropy-based Activation Function Optimization (EAFO) methodology. EAFO
methodology presents a novel perspective for designing static activation
functions in deep neural networks and the potential of dynamically optimizing
activation during iterative training. Utilizing EAFO methodology, we derive a
novel activation function from ReLU, known as Correction Regularized ReLU
(CRReLU). Experiments conducted with vision transformer and its variants on
CIFAR-10, CIFAR-100 and ImageNet-1K datasets demonstrate the superiority of
CRReLU over existing corrections of ReLU. Extensive empirical studies on task
of large language model (LLM) fine-tuning, CRReLU exhibits superior performance
compared to GELU, suggesting its broader potential for practical applications.

通过信息熵的角度，本研究理论上证明了存在具有边界条件的最差激活函数，提出了基于熵的激活函数优化方法（EAFO），并从 ReLU 中推导出了一种新的激活函数 CRReLU。实验证明 CRReLU 在深度神经网络中表现优异，并在大型语言模型细调任务中展现出与 GELU 相比的卓越性能，显示其广泛的实际应用潜力。