Despite the recent successes of deep neural networks, the corresponding
training problem remains highly non-convex and difficult to optimize. Classes
of models have been proposed that introduce greater structure to the objective
function at the cost of lifting the dimension of the problem. However, these
lifted methods sometimes perform poorly compared to traditional neural
networks. In this paper, we introduce a new class of lifted models, Fenchel
lifted networks, that enjoy the same benefits as previous lifted models,
without suffering a degradation in performance over classical networks. Our
model represents activation functions as equivalent biconvex constraints and
uses Lagrange Multipliers to arrive at a rigorous lower bound of the
traditional neural network training problem. This model is efficiently trained
using block-coordinate descent and is parallelizable across data points and/or
layers. We compare our model against standard fully connected and convolutional
networks and show that we are able to match or beat their performance.

本文介绍了一种新的 Fenchel lifted networks 的模型，通过等价的双凸约束表示激活函数，并使用拉格朗日乘数对传统神经网络训练问题得出严格下界，使用块坐标下降进行高效训练，并与标准全连接和卷积网络进行比较，证明我们能够匹配或超越其性能.