We study universal traits which emerge both in real-world complex datasets,
as well as in artificially generated ones. Our approach is to analogize data to
a physical system and employ tools from statistical physics and Random Matrix
Theory (RMT) to reveal their underlying structure. We focus on the
feature-feature covariance matrix, analyzing both its local and global
eigenvalue statistics. Our main observations are: (i) The power-law scalings
that the bulk of its eigenvalues exhibit are vastly different for uncorrelated
random data compared to real-world data, (ii) this scaling behavior can be
completely recovered by introducing long range correlations in a simple way to
the synthetic data, (iii) both generated and real-world datasets lie in the
same universality class from the RMT perspective, as chaotic rather than
integrable systems, (iv) the expected RMT statistical behavior already
manifests for empirical covariance matrices at dataset sizes significantly
smaller than those conventionally used for real-world training, and can be
related to the number of samples required to approximate the population
power-law scaling behavior, (v) the Shannon entropy is correlated with local
RMT structure and eigenvalues scaling, and substantially smaller in strongly
correlated datasets compared to uncorrelated synthetic data, and requires fewer
samples to reach the distribution entropy. These findings can have numerous
implications to the characterization of the complexity of data sets, including
differentiating synthetically generated from natural data, quantifying noise,
developing better data pruning methods and classifying effective learning
models utilizing these scaling laws.

本文使用统计物理学和随机矩阵理论，探究出现在真实和人造数据集中的普遍特征，发现特征 - 特征协方差矩阵的本地和全局特征值在数据规模方面存在着重要差异，而 Shannon 熵则与局部 RMT 结构和特征值尺度有关，并且与强相关数据集相比于无相关性的人造数据集要小很多。