Deep learning is currently extensively employed across a range of research
domains. The continuous advancements in deep learning techniques contribute to
solving intricate challenges. Activation functions (AF) are fundamental
components within neural networks, enabling them to capture complex patterns
and relationships in the data. By introducing non-linearities, AF empowers
neural networks to model and adapt to the diverse and nuanced nature of
real-world data, enhancing their ability to make accurate predictions across
various tasks. In the context of intrusion detection, the Mish, a recent AF,
was implemented in the CNN-BiGRU model, using three datasets: ASNM-TUN,
ASNM-CDX, and HOGZILLA. The comparison with Rectified Linear Unit (ReLU), a
widely used AF, revealed that Mish outperforms ReLU, showcasing superior
performance across the evaluated datasets. This study illuminates the
effectiveness of AF in elevating the performance of intrusion detection
systems.

该研究通过比较网络激活函数 Mish 和 ReLU 在入侵检测任务上的性能，揭示了激活函数在提升入侵检测系统性能方面的有效性。

优化 CNN-Bigru 性能：Mish 激活函数与 Relu 的对比分析

Optimizing cnn-Bigru performance: Mish activation and comparative  analysis with Relu

Activation Functions introduce non-linearity in the deep neural networks.
This nonlinearity helps the neural networks learn faster and efficiently from
the dataset. In deep learning, many activation functions are developed and used
based on the type of problem statement. ReLU's variants, SWISH, and MISH are
goto activation functions. MISH function is considered having similar or even
better performance than SWISH, and much better than ReLU. In this paper, we
propose an activation function named APTx which behaves similar to MISH, but
requires lesser mathematical operations to compute. The lesser computational
requirements of APTx does speed up the model training, and thus also reduces
the hardware requirement for the deep learning model.

该研究论文介绍了不同类型的激活函数对于深度学习的影响，提出了新的激活函数 APTx，其计算速度比 MISH 更快，但效果相近，同时可以有效减少深度学习的计算资源消耗。

APTx：比深度学习中使用的 MISH、SWISH 和 ReLU 变种更好的激活函数

APTx: better activation function than MISH, SWISH, and ReLU's variants used in deep learning

We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation
function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As
activation functions play a crucial role in the performance and training
dynamics in neural networks, we validated experimentally on several well-known
benchmarks against the best combinations of architectures and activation
functions. We also observe that data augmentation techniques have a favorable
effect on benchmarks like ImageNet-1k and MS-COCO across multiple
architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a
CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in
MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy
by $\approx$1$\%$ while keeping all other network parameters and
hyperparameters constant. Furthermore, we explore the mathematical formulation
of Mish in relation with the Swish family of functions and propose an intuitive
understanding on how the first derivative behavior may be acting as a
regularizer helping the optimization of deep neural networks. Code is publicly
available at this https URL

本研究提出了一种名为 Mish 的自正则非单调激活函数，其数学公式为：$f (x)=x	anh (softplus (x))$。我们通过在几个知名基准测试中验证证实该函数的有效性，并观察到数据增强技术对 ImageNet-1k 和 MS-COCO 等基准测试的积极作用。同时，我们还对 Mish 与 Swish 函数族的数学公式进行了探讨，并提出了一种直观的理解方式，即 Mish 的一阶导数行为可能作为一个正则化器帮助深度神经网络的优化，代码可在该网站公开获取。