Activation function is a pivotal component of deep learning, facilitating the
extraction of intricate data patterns. While classical activation functions
like ReLU and its variants are extensively utilized, their static nature and
simplicity, despite being advantageous, often limit their effectiveness in
specialized tasks. The trainable activation functions also struggle sometimes
to adapt to the unique characteristics of the data. Addressing these
limitations, we introduce a novel trainable activation function, adaptive
piecewise approximated activation linear unit (APALU), to enhance the learning
performance of deep learning across a broad range of tasks. It presents a
unique set of features that enable it to maintain stability and efficiency in
the learning process while adapting to complex data representations.
Experiments reveal significant improvements over widely used activation
functions for different tasks. In image classification, APALU increases
MobileNet and GoogleNet accuracy by 0.37% and 0.04%, respectively, on the
CIFAR10 dataset. In anomaly detection, it improves the average area under the
curve of One-CLASS Deep SVDD by 0.8% on the MNIST dataset, 1.81% and 1.11%
improvements with DifferNet, and knowledge distillation, respectively, on the
MVTech dataset. Notably, APALU achieves 100% accuracy on a sign language
recognition task with a limited dataset. For regression tasks, APALU enhances
the performance of deep neural networks and recurrent neural networks on
different datasets. These improvements highlight the robustness and
adaptability of APALU across diverse deep-learning applications.

训练激活函数 APALU 在深度学习中具有鲁棒性和适应性，通过提升深度神经网络和递归神经网络在各种任务中的学习性能，以及在图像分类、异常检测和回归任务中广泛应用的激活函数的改进，证明其在不同的深度学习应用中表现出稳定性和效率的特点。