We describe a novel family of models of multi- layer feedforward neural
networks in which the activation functions are encoded via penalties in the
training problem. Our approach is based on representing a non-decreasing
activation function as the argmin of an appropriate convex optimiza- tion
problem. The new framework allows for algo- rithms such as block-coordinate
descent methods to be applied, in which each step is composed of a simple (no
hidden layer) supervised learning problem that is parallelizable across data
points and/or layers. Experiments indicate that the pro- posed models provide
excellent initial guesses for weights for standard neural networks. In addi-
tion, the model provides avenues for interesting extensions, such as robustness
against noisy in- puts and optimizing over parameters in activation functions.

本文提出一种新型的神经网络构架，利用基于惩罚项的训练问题来编码激活函数，这种框架可以被应用于 block-coordinate descent 算法中，该算法可以在每次迭代中通过并行化数据点和 / 或层数来解决简单（没有隐藏层）的监督学习问题，实验结果表明该方法为标准神经网络提供了极佳的初始权重估计，并且对于使用参数优化激活函数、对抗噪声数据的拓展也提供了思路。