This paper proposes a learnable nonlinear activation mechanism specifically
for convolutional neural network (CNN) termed as LENI, which learns to enhance
the negative information in CNNs. In sharp contrast to ReLU which cuts off the
negative neurons and suffers from the issue of ''dying ReLU'', LENI enjoys the
capacity to reconstruct the dead neurons and reduce the information loss.
Compared to improved ReLUs, LENI introduces a learnable approach to process the
negative phase information more properly. In this way, LENI can enhance the
model representational capacity significantly while maintaining the original
advantages of ReLU. As a generic activation mechanism, LENI possesses the
property of portability and can be easily utilized in any CNN models through
simply replacing the activation layers with LENI block. Extensive experiments
validate that LENI can improve the performance of various baseline models on
various benchmark datasets by a clear margin (up to 1.24% higher top-1 accuracy
on ImageNet-1k) with negligible extra parameters. Further experiments show that
LENI can act as a channel compensation mechanism, offering competitive or even
better performance but with fewer learned parameters than baseline models. In
addition, LENI introduces the asymmetry to the model structure which
contributes to the enhancement of representational capacity. Through
visualization experiments, we validate that LENI can retain more information
and learn more representations.

本文提出了一种可学习的非线性激活机制 LENI，旨在增强 CNN 中的负信息，并通过实验证明 LENI 可以提高模型的表示能力，同时保持 ReLU 的优点。

学习如何增强卷积神经网络中的负面信息

Learn to Enhance the Negative Information in Convolutional Neural  Network

We study the fundamental problems of agnostically learning halfspaces and
ReLUs under Gaussian marginals. In the former problem, given labeled examples
$(\mathbf{x}, y)$ from an unknown distribution on $\mathbb{R}^d \times \{ \pm
1\}$, whose marginal distribution on $\mathbf{x}$ is the standard Gaussian and
the labels $y$ can be arbitrary, the goal is to output a hypothesis with 0-1
loss $\mathrm{OPT}+\epsilon$, where $\mathrm{OPT}$ is the 0-1 loss of the
best-fitting halfspace. In the latter problem, given labeled examples
$(\mathbf{x}, y)$ from an unknown distribution on $\mathbb{R}^d \times
\mathbb{R}$, whose marginal distribution on $\mathbf{x}$ is the standard
Gaussian and the labels $y$ can be arbitrary, the goal is to output a
hypothesis with square loss $\mathrm{OPT}+\epsilon$, where $\mathrm{OPT}$ is
the square loss of the best-fitting ReLU. We prove Statistical Query (SQ) lower
bounds of $d^{\mathrm{poly}(1/\epsilon)}$ for both of these problems. Our SQ
lower bounds provide strong evidence that current upper bounds for these tasks
are essentially best possible.

该论文研究了在高斯边际下以不可知方式学习半空间和 ReLU 的基本问题，证明了这些问题的统计查询下限。

高斯边缘下分别学习半空间和 ReLU 的近最优 SQ 下界

Near-Optimal SQ Lower Bounds for Agnostically Learning Halfspaces and  ReLUs under Gaussian Marginals

Rectifier neuron units (ReLUs) have been widely used in deep convolutional
networks. An ReLU converts negative values to zeros, and does not change
positive values, which leads to a high sparsity of neurons. In this work, we
first examine the sparsity of the outputs of ReLUs in some popular deep
convolutional architectures. And then we use the sparsity property of ReLUs to
accelerate the calculation of convolution by skipping calculations of
zero-valued neurons. The proposed sparse convolution algorithm achieves some
speedup improvements on CPUs compared to the traditional matrix-matrix
multiplication algorithm for convolution when the sparsity is not less than
0.9.

本研究针对深度卷积神经网络中常用的整流线性单元（ReLU），探究其输出的稀疏性质，并借此来加速卷积计算，通过跳过计算零值神经元的方法，相比传统矩阵乘法，当稀疏度不低于 0.9 时，在 CPU 上实现了较大的加速。