Threshold activation functions are highly preferable in neural networks due
to their efficiency in hardware implementations. Moreover, their mode of
operation is more interpretable and resembles that of biological neurons.
However, traditional gradient based algorithms such as Gradient Descent cannot
be used to train the parameters of neural networks with threshold activations
since the activation function has zero gradient except at a single
non-differentiable point. To this end, we study weight decay regularized
training problems of deep neural networks with threshold activations. We first
show that regularized deep threshold network training problems can be
equivalently formulated as a standard convex optimization problem, which
parallels the LASSO method, provided that the last hidden layer width exceeds a
certain threshold. We also derive a simplified convex optimization formulation
when the dataset can be shattered at a certain layer of the network. We
corroborate our theoretical results with various numerical experiments.

本文研究了具有阈值激活函数的神经网络的权重衰减正则化训练问题，发现在网络的最后一个隐藏层宽度超过一定阈值时，这些问题可以被等价地形式化为标准的凸优化问题，类似于 LASSO 方法，并且通过各种数字实验验证理论结果。