This paper proposes a learnable nonlinear activation mechanism specifically
for convolutional neural network (CNN) termed as LENI, which learns to enhance
the negative information in CNNs. In sharp contrast to ReLU which cuts off the
negative neurons and suffers from the issue of ''dying ReLU'', LENI enjoys the
capacity to reconstruct the dead neurons and reduce the information loss.
Compared to improved ReLUs, LENI introduces a learnable approach to process the
negative phase information more properly. In this way, LENI can enhance the
model representational capacity significantly while maintaining the original
advantages of ReLU. As a generic activation mechanism, LENI possesses the
property of portability and can be easily utilized in any CNN models through
simply replacing the activation layers with LENI block. Extensive experiments
validate that LENI can improve the performance of various baseline models on
various benchmark datasets by a clear margin (up to 1.24% higher top-1 accuracy
on ImageNet-1k) with negligible extra parameters. Further experiments show that
LENI can act as a channel compensation mechanism, offering competitive or even
better performance but with fewer learned parameters than baseline models. In
addition, LENI introduces the asymmetry to the model structure which
contributes to the enhancement of representational capacity. Through
visualization experiments, we validate that LENI can retain more information
and learn more representations.

本文提出了一种可学习的非线性激活机制 LENI，旨在增强 CNN 中的负信息，并通过实验证明 LENI 可以提高模型的表示能力，同时保持 ReLU 的优点。

学习如何增强卷积神经网络中的负面信息

Learn to Enhance the Negative Information in Convolutional Neural  Network

In this work, we propose a novel activation mechanism aimed at establishing
layer-level activation (LayerAct) functions. These functions are designed to be
more noise-robust compared to traditional element-level activation functions by
reducing the layer-level fluctuation of the activation outputs due to shift in
inputs. Moreover, the LayerAct functions achieve a zero-like mean activation
output without restricting the activation output space. We present an analysis
and experiments demonstrating that LayerAct functions exhibit superior
noise-robustness compared to element-level activation functions, and
empirically show that these functions have a zero-like mean activation.
Experimental results on three benchmark image classification tasks show that
LayerAct functions excel in handling noisy image datasets, outperforming
element-level activation functions, while the performance on clean datasets is
also superior in most cases.

本文提出了一种新颖的激活机制，名为 LayerAct 函数，该函数能够更好地应对输入的波动和噪声，同时在保持零输出均值的同时，实现比传统的元素级激活函数更好的噪声鲁棒性，实验结果表明，这些函数在处理噪声图像数据集方面表现出色，同时在大多数情况下，对于清晰的图像数据集的性能也更优秀。