The distributional simplicity bias (DSB) posits that neural networks learn
low-order moments of the data distribution first, before moving on to
higher-order correlations. In this work, we present compelling new evidence for
the DSB by showing that networks automatically learn to perform well on
maximum-entropy distributions whose low-order statistics match those of the
training set early in training, then lose this ability later. We also extend
the DSB to discrete domains by proving an equivalence between token $n$-gram
frequencies and the moments of embedding vectors, and by finding empirical
evidence for the bias in LLMs. Finally we use optimal transport methods to
surgically edit the low-order statistics of one class to match those of
another, and show that early-training networks treat the edited samples as if
they were drawn from the target class. Code is available at
this https URL

该研究指出神经网络首先学习数据分布的低阶矩，然后再学习高阶相关性。通过实验证据，证明了神经网络在训练早期会自动学习以最大熵分布为基础的低阶统计信息，但训练后期会失去这种能力。该研究还将分布简易偏差扩展到离散领域，并通过证明 Token 的 n-gram 频率与嵌入向量矩的等价性以及在 LLMs 中找到的实证证据，证明了该偏差。最后，研究利用最优运输方法对某一类的低阶统计信息进行手术式编辑，使其与目标类的统计信息相匹配，并展示了早期训练的网络将编辑后的样本视为从目标类中抽取的样本。

神经网络学习逐渐复杂的统计规律

Neural Networks Learn Statistics of Increasing Complexity

This work studies applications and generalizations of a simple estimation
technique that provides exponential concentration under heavy-tailed
distributions, assuming only bounded low-order moments. We show that the
technique can be used for approximate minimization of smooth and strongly
convex losses, and specifically for least squares linear regression. For
instance, our $d$-dimensional estimator requires just
$\tilde{O}(d\log(1/\delta))$ random samples to obtain a constant factor
approximation to the optimal least squares loss with probability $1-\delta$,
without requiring the covariates or noise to be bounded or subgaussian. We
provide further applications to sparse linear regression and low-rank
covariance matrix estimation with similar allowances on the noise and covariate
distributions. The core technique is a generalization of the median-of-means
estimator to arbitrary metric spaces.

该论文研究了一种简单估计技术在重尾分布下提供指数集中性的应用和推广，证明该技术可用于平滑强凸损失函数的近似最小化，特别是在最小二乘线性回归、稀疏线性回归和低秩协方差矩阵估计中具有类似的特征。