While much progress has been achieved over the last decades in neuro-inspired
machine learning, there are still fundamental theoretical problems in
gradient-based learning using combinations of neurons. These problems, such as
saddle points and suboptimal plateaus of the cost function, can lead in theory
and practice to failures of learning. In addition, the discrete step size
selection of the gradient is problematic since too large steps can lead to
instability and too small steps slow down the learning. This paper describes an
alternative discrete MinMax learning approach for continuous piece-wise linear
functions. Global exponential convergence of the algorithm is established using
Contraction Theory with Inequality Constraints, which is extended from the
continuous to the discrete case in this paper:
The parametrization of each linear function piece is, in contrast to deep
learning, linear in the proposed MinMax network. This allows a linear
regression stability proof as long as measurements do not transit from one
linear region to its neighbouring linear region.
The step size of the discrete gradient descent is Lagrangian limited
orthogonal to the edge of two neighbouring linear functions. It will be shown
that this Lagrangian step limitation does not decrease the convergence of the
unconstrained system dynamics in contrast to a step size limitation in the
direction of the gradient.
We show that the convergence rate of a constrained piece-wise linear function
learning is equivalent to the exponential convergence rates of the individual
local linear regions.

本文描述了一种替代梯度下降算法的离散 MinMax 学习方法，通过使用缩减理论，将该算法从连续情况扩展到离散情况。通过该方法，解决了基于神经元组合的梯度下降学习中的最优性问题和成本函数问题。