By lifting the ReLU function into a higher dimensional space, we develop a
smooth multi-convex formulation for training feed-forward deep neural networks
(DNNs). This allows us to develop a block coordinate descent (BCD) training
algorithm consisting of a sequence of numerically well-behaved convex
optimizations. Using ideas from proximal point methods in convex analysis, we
prove that this BCD algorithm will converge globally to a stationary point with
R-linear convergence rate of order one. In experiments with the MNIST database,
DNNs trained with this BCD algorithm consistently yielded better test-set error
rates than identical DNN architectures trained via all the stochastic gradient
descent (SGD) variants in the Caffe toolbox.

本文提出了一种用于训练深度神经网络的光滑的多凸形式，该方法利用了凸分析中的近端点方法，开发了一个块协调下降（BCD）训练算法，证明了其具有全局收敛性和 R - 线性收敛速率，并在实验中展示了优于 Caffe 工具箱中所有随机梯度下降（SGD）变体的表现。

用于训练 Tikhonov 正则化深度神经网络的收敛块坐标下降算法

Convergent Block Coordinate Descent for Training Tikhonov Regularized  Deep Neural Networks

The paper briefy reviews several recent results on hierarchical architectures
for learning from examples, that may formally explain the conditions under
which Deep Convolutional Neural Networks perform much better in function
approximation problems than shallow, one-hidden layer architectures. The paper
announces new results for a non-smooth activation function - the ReLU function
- used in present-day neural networks, as well as for the Gaussian networks. We
propose a new definition of relative dimension to encapsulate different notions
of sparsity of a function class that can possibly be exploited by deep networks
but not by shallow ones to drastically reduce the complexity required for
approximation and learning.

本文回顾了最近关于层级神经网络结构的研究成果，探讨了深度卷积神经网络优于浅层神经网络在函数近似问题中的表现条件。本文提出了一个新的对于相对维度的定义，该定义可以被深层网络而非浅层网络使用以显著降低近似和学习所需的复杂度。同时，本文还宣布了关于当前神经网络中使用的非平滑激活函数 - ReLU 函数以及高斯网络的新结果。