Matrix factorization models have been extensively studied as a valuable
test-bed for understanding the implicit biases of overparameterized models.
Although both low nuclear norm and low rank regularization have been studied
for these models, a unified understanding of when, how, and why they achieve
different implicit regularization effects remains elusive. In this work, we
systematically investigate the implicit regularization of matrix factorization
for solving matrix completion problems. We empirically discover that the
connectivity of observed data plays a crucial role in the implicit bias, with a
transition from low nuclear norm to low rank as data shifts from disconnected
to connected with increased observations. We identify a hierarchy of intrinsic
invariant manifolds in the loss landscape that guide the training trajectory to
evolve from low-rank to higher-rank solutions. Based on this finding, we
theoretically characterize the training trajectory as following the
hierarchical invariant manifold traversal process, generalizing the
characterization of Li et al. (2020) to include the disconnected case.
Furthermore, we establish conditions that guarantee minimum nuclear norm,
closely aligning with our experimental findings, and we provide a dynamics
characterization condition for ensuring minimum rank. Our work reveals the
intricate interplay between data connectivity, training dynamics, and implicit
regularization in matrix factorization models.

矩阵分解模型在理解超参数模型的隐含偏差方面，作为一个重要的测试平台已被广泛研究。本文系统地研究了矩阵分解模型在解决矩阵补全问题时的隐含正则化，通过实验证明观测数据的连接性在隐含偏差中起着关键作用，其随着观察次数的增加由低核范数转变为低秩解。我们发现损失函数引导训练轨迹从低秩解演化到更高秩解的过程在损失函数的损失面内有一系列内在的不变流行，并基于此给出了训练轨迹的理论刻画，进一步建立了最小核范数的条件，与实验结果相吻合，同时为保证最小秩解提供了动力学刻画条件。我们的工作揭示了矩阵分解模型中数据连接性、训练动力学和隐含正则化之间复杂的相互作用。

连接性塑造矩阵完备性模型的隐式正则化

Connectivity Shapes Implicit Regularization in Matrix Factorization  Models for Matrix Completion

We consider the case of inpainting single depth images. Without corresponding
color images, previous or next frames, depth image inpainting is quite
challenging. One natural solution is to regard the image as a matrix and adopt
the low rank regularization just as inpainting color images. However, the low
rank assumption does not make full use of the properties of depth images.
A shallow observation may inspire us to penalize the non-zero gradients by
sparse gradient regularization. However, statistics show that though most
pixels have zero gradients, there is still a non-ignorable part of pixels whose
gradients are equal to 1. Based on this specific property of depth images , we
propose a low gradient regularization method in which we reduce the penalty for
gradient 1 while penalizing the non-zero gradients to allow for gradual depth
changes. The proposed low gradient regularization is integrated with the low
rank regularization into the low rank low gradient approach for depth image
inpainting. We compare our proposed low gradient regularization with sparse
gradient regularization. The experimental results show the effectiveness of our
proposed approach.

本研究提出了一种针对深度图像修复的新的低梯度正则化方法，试图更好地利用深度图像的属性，将其与低秩正则化相结合，得到了实验效果比传统的稀疏梯度正则化更好的低秩低梯度方法。