The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the
study of wide Neural Networks. In particular, it is known that the positivity
of the NTK is directly related to the memorization capacity of sufficiently
wide networks, i.e., to the possibility of reaching zero loss in training, via
gradient descent. Here we will improve on previous works and obtain a sharp
result concerning the positivity of the NTK of feedforward networks of any
depth. More precisely, we will show that, for any non-polynomial activation
function, the NTK is strictly positive definite. Our results are based on a
novel characterization of polynomial functions which is of independent
interest.

对于任何非多项式激活函数，神经切向核（NTK）是严格正定的，这个结果与宽神经网络的记忆能力直接相关。

神经切向核的正性

The Positivity of the Neural Tangent Kernel

Variational regularization is commonly used to solve linear inverse problems,
and involves augmenting a data fidelity by a regularizer. The regularizer is
used to promote a priori information, and is weighted by a regularization
parameter. Selection of an appropriate regularization parameter is critical,
with various choices leading to very different reconstructions. Existing
strategies such as the discrepancy principle and L-curve can be used to
determine a suitable parameter value, but in recent years a supervised machine
learning approach called bilevel learning has been employed. Bilevel learning
is a powerful framework to determine optimal parameters, and involves solving a
nested optimisation problem. While previous strategies enjoy various
theoretical results, the well-posedness of bilevel learning in this setting is
still a developing field. One necessary property is positivity of the
determined regularization parameter. In this work, we provide a new condition
that better characterises positivity of optimal regularization parameters than
the existing theory. Numerical results verify and explore this new condition
for both small and large dimensional problems.

利用变分正则化方法求解线性反问题时，采用超参数调节正则化项，通过强化先验信息来提高求解结果，核心问题在于如何选择一个合适的正则化参数。本文提出了一个新的条件来更好地描述正则化参数的正性，并验证和探索了这个新条件，同时也探究了这个新条件在小和大维问题上的应用。