State-of-the-art neural network training methods depend on the gradient of
the network function. Therefore, they cannot be applied to networks whose
activation functions do not have useful derivatives, such as binary and
discrete-time spiking neural networks. To overcome this problem, the activation
function's derivative is commonly substituted with a surrogate derivative,
giving rise to surrogate gradient learning (SGL). This method works well in
practice but lacks theoretical foundation. The neural tangent kernel (NTK) has
proven successful in the analysis of gradient descent. Here, we provide a
generalization of the NTK, which we call the surrogate gradient NTK, that
enables the analysis of SGL. First, we study a naive extension of the NTK to
activation functions with jumps, demonstrating that gradient descent for such
activation functions is also ill-posed in the infinite-width limit. To address
this problem, we generalize the NTK to gradient descent with surrogate
derivatives, i.e., SGL. We carefully define this generalization and expand the
existing key theorems on the NTK with mathematical rigor. Further, we
illustrate our findings with numerical experiments. Finally, we numerically
compare SGL in networks with sign activation function and finite width to
kernel regression with the surrogate gradient NTK; the results confirm that the
surrogate gradient NTK provides a good characterization of SGL.

研究了神经网络训练方法中激活函数导数不可用时的问题，提出了代理梯度学习（SGL）的理论基础，并利用神经切向核（NTK）的推广 —— 代理梯度 NTK 分析了 SGL，通过数值实验验证了 SGL 在具有有限宽度和符号激活函数的网络中的有效性。