We study the implicit bias of gradient flow (i.e., gradient descent with
infinitesimal step size) on linear neural network training. We propose a tensor
formulation of neural networks that includes fully-connected, diagonal, and
convolutional networks as special cases, and investigate the linear version of
the formulation called linear tensor networks. With this formulation, we can
characterize the convergence direction of the network parameters as singular
vectors of a tensor defined by the network. For $L$-layer linear tensor
networks that are orthogonally decomposable, we show that gradient flow on
separable classification finds a stationary point of the $\ell_{2/L}$
max-margin problem in a "transformed" input space defined by the network. For
underdetermined regression, we prove that gradient flow finds a global minimum
which minimizes a norm-like function that interpolates between weighted
$\ell_1$ and $\ell_2$ norms in the transformed input space. Our theorems
subsume existing results in the literature while removing standard convergence
assumptions. We also provide experiments that corroborate our analysis.

研究了线性神经网络训练中渐进流（即用无穷小步长的梯度下降法）的隐含偏差；提出了神经网络的张量形式，包括全连接、对角线和卷积网络等特例，并研究了称为线性张量网络的公式的线性版本。通过这个公式，我们可以将网络的收敛方向表征为由网络定义的张量的奇异向量。