Matrix factorization models have been extensively studied as a valuable
test-bed for understanding the implicit biases of overparameterized models.
Although both low nuclear norm and low rank regularization have been studied
for these models, a unified understanding of when, how, and why they achieve
different implicit regularization effects remains elusive. In this work, we
systematically investigate the implicit regularization of matrix factorization
for solving matrix completion problems. We empirically discover that the
connectivity of observed data plays a crucial role in the implicit bias, with a
transition from low nuclear norm to low rank as data shifts from disconnected
to connected with increased observations. We identify a hierarchy of intrinsic
invariant manifolds in the loss landscape that guide the training trajectory to
evolve from low-rank to higher-rank solutions. Based on this finding, we
theoretically characterize the training trajectory as following the
hierarchical invariant manifold traversal process, generalizing the
characterization of Li et al. (2020) to include the disconnected case.
Furthermore, we establish conditions that guarantee minimum nuclear norm,
closely aligning with our experimental findings, and we provide a dynamics
characterization condition for ensuring minimum rank. Our work reveals the
intricate interplay between data connectivity, training dynamics, and implicit
regularization in matrix factorization models.

矩阵分解模型在理解超参数模型的隐含偏差方面，作为一个重要的测试平台已被广泛研究。本文系统地研究了矩阵分解模型在解决矩阵补全问题时的隐含正则化，通过实验证明观测数据的连接性在隐含偏差中起着关键作用，其随着观察次数的增加由低核范数转变为低秩解。我们发现损失函数引导训练轨迹从低秩解演化到更高秩解的过程在损失函数的损失面内有一系列内在的不变流行，并基于此给出了训练轨迹的理论刻画，进一步建立了最小核范数的条件，与实验结果相吻合，同时为保证最小秩解提供了动力学刻画条件。我们的工作揭示了矩阵分解模型中数据连接性、训练动力学和隐含正则化之间复杂的相互作用。