Numerous researchers recently applied empirical spectral analysis to the
study of modern deep learning classifiers. We identify and discuss an important
formal class/cross-class structure and show how it lies at the origin of the
many visually striking features observed in deepnet spectra, some of which were
reported in recent articles, others are unveiled here for the first time. These
include spectral outliers, "spikes", and small but distinct continuous
distributions, "bumps", often seen beyond the edge of a "main bulk".
The significance of the cross-class structure is illustrated in three ways:
(i) we prove the ratio of outliers to bulk in the spectrum of the Fisher
information matrix is predictive of misclassification, in the context of
multinomial logistic regression; (ii) we demonstrate how, gradually with depth,
a network is able to separate class-distinctive information from class
variability, all while orthogonalizing the class-distinctive information; and
(iii) we propose a correction to KFAC, a well-known second-order optimization
algorithm for training deepnets.

研究应用实证频谱分析于现代深度学习分类器，展示其在深度网络频谱中的重要形式类 / 跨类结构及其特征，包括频谱异常值、突起和小但清晰的连续分布，介绍跨类结构的重要性，并证明多项式逻辑回归中，Fisher 信息矩阵频谱中异常值与主体审查的比例预测了误分类，在深度上，网络能够在将类别区分信息正交化的同时，逐渐分离类别特定信息，并提出了 KFAC 的校正。