The phenomenon of implicit regularization has attracted interest in recent
years as a fundamental aspect of the remarkable generalizing ability of neural
networks. In a nutshell, it entails that gradient descent dynamics in many
neural nets, even without any explicit regularizer in the loss function,
converges to the solution of a regularized learning problem. However, known
results attempting to theoretically explain this phenomenon focus
overwhelmingly on the setting of linear neural nets, and the simplicity of the
linear structure is particularly crucial to existing arguments. In this paper,
we explore this problem in the context of more realistic neural networks with a
general class of non-linear activation functions, and rigorously demonstrate
the implicit regularization phenomenon for such networks in the setting of
matrix sensing problems, together with rigorous rate guarantees that ensure
exponentially fast convergence of gradient descent.In this vein, we contribute
a network architecture called Spectral Neural Networks (abbrv. SNN) that is
particularly suitable for matrix learning problems. Conceptually, this entails
coordinatizing the space of matrices by their singular values and singular
vectors, as opposed to by their entries, a potentially fruitful perspective for
matrix learning. We demonstrate that the SNN architecture is inherently much
more amenable to theoretical analysis than vanilla neural nets and confirm its
effectiveness in the context of matrix sensing, via both mathematical
guarantees and empirical investigations. We believe that the SNN architecture
has the potential to be of wide applicability in a broad class of matrix
learning scenarios.

通过对带有非线性激活函数的神经网络在矩阵感知问题中的隐性正则化现象的研究，我们引入了一种名为 “光谱神经网络（SNN）” 的网络架构，该架构在矩阵学习问题上具有更好的可解释性，通过数学保证和实证研究验证了其有效性。我们相信 SNN 架构在广泛的矩阵学习场景中具有潜在的适用性。