Lightweight or mobile neural networks used for real-time computer vision
tasks contain fewer parameters than normal networks, which lead to a
constrained performance. In this work, we proposed a novel activation function
named Tanh Exponential Activation Function (TanhExp) which can improve the
performance for these networks on image classification task significantly. The
definition of TanhExp is f(x) = xtanh(e^x). We demonstrate the simplicity,
efficiency, and robustness of TanhExp on various datasets and network models
and TanhExp outperforms its counterparts in both convergence speed and
accuracy. Its behaviour also remains stable even with noise added and dataset
altered. We show that without increasing the size of the network, the capacity
of lightweight neural networks can be enhanced by TanhExp with only a few
training epochs and no extra parameters added.

在轻量级神经网络领域提出了一种名为 TanhExponential 激活函数，可以有效提高无参数增加的网络的图像分类性能。通过不断试验，证明了 TanhExponential 函数对于各种数据集和网络模型的效率，稳定性、收敛速度和准确性优于其它函数。

TanhExp：一种平滑的激活函数，具有高收敛速度，适用于轻量级神经网络

TanhExp: A Smooth Activation Function with High Convergence Speed for  Lightweight Neural Networks

Many applications of mobile deep learning, especially real-time computer
vision workloads, are constrained by computation power. This is particularly
true for workloads running on older consumer phones, where a typical device
might be powered by a single- or dual-core ARMv7 CPU. We provide an open-source
implementation and a comprehensive analysis of (to our knowledge) the state of
the art ultra-low-precision (<4 bit precision) implementation of the core
primitives required for modern deep learning workloads on ARMv7 devices, and
demonstrate speedups of 4x-20x over our additional state-of-the-art float32 and
int8 baselines.

通过对 ARMv7 设备上现代深度学习工作负载所需的核心基本操作进行开源实现和全面分析，我们展示了与现有市场上的 float32 和 int8 基准相比，使用大于 4 位精度的最先进的超低精度技术可获得 4 倍至 20 倍的加速度。