Deep learning training training algorithms are a huge success in recent years
in many fields including speech, text,image video etc. Deeper and deeper layers
are proposed with huge success with resnet structures having around 152 layers.
Shallow convolution neural networks(CNN's) are still an active research, where
some phenomena are still unexplained. Activation functions used in the network
are of utmost importance, as they provide non linearity to the networks. Relu's
are the most commonly used activation function.We show a complex piece-wise
linear(PWL) activation in the hidden layer. We show that these PWL activations
work much better than relu activations in our networks for convolution neural
networks and multilayer perceptrons. Result comparison in PyTorch for shallow
and deep CNNs are given to further strengthen our case.

复杂的分段线性激活函数在浅层和深层卷积神经网络中比 ReLu 激活函数效果更好，并使用 PyTorch 进行结果比较。

通过动态激活函数优化前馈和卷积神经网络的性能

Optimizing Performance of Feedforward and Convolutional Neural Networks  through Dynamic Activation Functions

We present an approach for the verification of feed-forward neural networks
in which all nodes have a piece-wise linear activation function. Such networks
are often used in deep learning and have been shown to be hard to verify for
modern satisfiability modulo theory (SMT) and integer linear programming (ILP)
solvers.
The starting point of our approach is the addition of a global linear
approximation of the overall network behavior to the verification problem that
helps with SMT-like reasoning over the network behavior. We present a
specialized verification algorithm that employs this approximation in a search
process in which it infers additional node phases for the non-linear nodes in
the network from partial node phase assignments, similar to unit propagation in
classical SAT solving. We also show how to infer additional conflict clauses
and safe node fixtures from the results of the analysis steps performed during
the search. The resulting approach is evaluated on collision avoidance and
handwritten digit recognition case studies.

本文提出了一种针对具备分段线性激活函数的前向神经网络进行验证的方法，并使用全局线性逼近对 SMT 求解器和 ILP 求解器进行推理，通过专门设计的验证算法利用推理出的相位信息对网络进行冲突检测和安全节点约束，并在碰撞避免和手写数字识别案例研究中进行了评价。