The Fisher information matrix (FIM) is a fundamental quantity to represent
the characteristics of a stochastic model, including deep neural networks
(DNNs). The present study reveals novel statistics of FIM that are universal
among a wide class of DNNs. To this end, we use random weights and large width
limits, which enables us to utilize mean field theories. We investigate the
asymptotic statistics of the FIM's eigenvalues and reveal that most of them are
close to zero while the maximum eigenvalue takes a huge value. Because the
landscape of the parameter space is defined by the FIM, it is locally flat in
most dimensions, but strongly distorted in others. Moreover, we demonstrate the
potential usage of the derived statistics in learning strategies. First, small
eigenvalues that induce flatness can be connected to a norm-based capacity
measure of generalization ability. Second, the maximum eigenvalue that induces
the distortion enables us to quantitatively estimate an appropriately sized
learning rate for gradient methods to converge.

研究发现：在计算深度神经网络（DNNs）的特性时，用 Fisher 信息矩阵（FIM）是一个普遍存在的策略。我们使用随机权重和大宽度限制来研究 FIM 的渐近统计特性，并证明说大部分特征值接近于零，而其中的最大特征值则取一个巨大的值。此外，我们探讨了这些新的统计特征数字在学习策略中的潜在使用，即可以量化估算出适当的学习率和我们的广义能力测量的基础。

深度神经网络中费舍尔信息的普适统计：平均场方法

Universal Statistics of Fisher Information in Deep Neural Networks: Mean  Field Approach

The Generalized Second Price (GSP) auction is the primary auction used for
monetizing the use of the Internet. It is well-known that truthtelling is not a
dominant strategy in this auction and that inefficient equilibria can arise. In
this paper we study the space of equilibria in GSP, and quantify the efficiency
loss that can arise in equilibria under a wide range of sources of uncertainty,
as well as in the full information setting. The traditional Bayesian game
models uncertainty in the valuations (types) of the participants. The
Generalized Second Price (GSP) auction gives rise to a further form of
uncertainty: the selection of quality factors resulting in uncertainty about
the behavior of the underlying ad allocation algorithm. The bounds we obtain
apply to both forms of uncertainty, and are robust in the sense that they apply
under various perturbations of the solution concept, extending to models with
information asymmetries and bounded rationality in the form of learning
strategies.
We present a constant bound (2.927) on the factor of the efficiency loss
(\emph{price of anarchy}) of the corresponding game for the Bayesian model of
partial information about other participants and about ad quality factors. For
the full information setting, we prove a surprisingly low upper bound of 1.282
on the price of anarchy over pure Nash equilibria, nearly matching a lower
bound of 1.259 for the case of three advertisers. Further, we do not require
that the system reaches equilibrium, and give similarly low bounds also on the
quality degradation for any no-regret learning outcome. Our conclusion is that
the number of advertisers in the auction has almost no impact on the price of
anarchy, and that the efficiency of GSP is very robust with respect to the
belief and rationality assumptions imposed on the participants.

本文研究了广义第二价格 (GSP) 拍卖中的均衡状态空间，并量化了广泛来源的不确定性和完全信息设置下可能出现的低效均衡状态损失。通过贝叶斯博弈模型，得出了 2.927 的损失因子上限，证明了拍卖参与者关于其他参与者和广告质量因素的部分信息的贝叶斯模型中的效率损失。此外，在完全信息设置中，我们证明了对于三个广告商的情况下，纯纳什均衡的 “惩罚因子” 的上限接近于 1.282，同时我们展示了类似的低上限的结果适用于任何遵循无后悔策略的拍卖结果。