We describe the convex semi-infinite dual of the two-layer vector-output ReLU
neural network training problem. This semi-infinite dual admits a finite
dimensional representation, but its support is over a convex set which is
difficult to characterize. In particular, we demonstrate that the non-convex
neural network training problem is equivalent to a finite-dimensional convex
copositive program. Our work is the first to identify this strong connection
between the global optima of neural networks and those of copositive programs.
We thus demonstrate how neural networks implicitly attempt to solve copositive
programs via semi-nonnegative matrix factorization, and draw key insights from
this formulation. We describe the first algorithms for provably finding the
global minimum of the vector output neural network training problem, which are
polynomial in the number of samples for a fixed data rank, yet exponential in
the dimension. However, in the case of convolutional architectures, the
computational complexity is exponential in only the filter size and polynomial
in all other parameters. We describe the circumstances in which we can find the
global optimum of this neural network training problem exactly with
soft-thresholded SVD, and provide a copositive relaxation which is guaranteed
to be exact for certain classes of problems, and which corresponds with the
solution of Stochastic Gradient Descent in practice.

这篇论文描述了两层向量输出 ReLU 神经网络训练问题的凸半无限对偶问题，利用它可以找到全局最优解，而神经网络隐式地尝试通过半非负矩阵分解解决共正程序。