The distributional simplicity bias (DSB) posits that neural networks learn
low-order moments of the data distribution first, before moving on to
higher-order correlations. In this work, we present compelling new evidence for
the DSB by showing that networks automatically learn to perform well on
maximum-entropy distributions whose low-order statistics match those of the
training set early in training, then lose this ability later. We also extend
the DSB to discrete domains by proving an equivalence between token $n$-gram
frequencies and the moments of embedding vectors, and by finding empirical
evidence for the bias in LLMs. Finally we use optimal transport methods to
surgically edit the low-order statistics of one class to match those of
another, and show that early-training networks treat the edited samples as if
they were drawn from the target class. Code is available at
this https URL

该研究指出神经网络首先学习数据分布的低阶矩，然后再学习高阶相关性。通过实验证据，证明了神经网络在训练早期会自动学习以最大熵分布为基础的低阶统计信息，但训练后期会失去这种能力。该研究还将分布简易偏差扩展到离散领域，并通过证明 Token 的 n-gram 频率与嵌入向量矩的等价性以及在 LLMs 中找到的实证证据，证明了该偏差。最后，研究利用最优运输方法对某一类的低阶统计信息进行手术式编辑，使其与目标类的统计信息相匹配，并展示了早期训练的网络将编辑后的样本视为从目标类中抽取的样本。

神经网络学习逐渐复杂的统计规律

Neural Networks Learn Statistics of Increasing Complexity

Maximum-Entropy Distributions offer an attractive family of probability
densities suitable for moment closure problems. Yet finding the Lagrange
multipliers which parametrize these distributions, turns out to be a
computational bottleneck for practical closure settings. Motivated by recent
success of Gaussian processes, we investigate the suitability of Gaussian
priors to approximate the Lagrange multipliers as a map of a given set of
moments. Examining various kernel functions, the hyperparameters are optimized
by maximizing the log-likelihood. The performance of the devised data-driven
Maximum-Entropy closure is studied for couple of test cases including
relaxation of non-equilibrium distributions governed by Bhatnagar-Gross-Krook
and Boltzmann kinetic equations.

最大熵分布、拉格朗日乘子、高斯过程、核函数和数据驱动的最大熵闭合问题的性能研究。