We present a novel theory of the microstructure of deep neural networks.
Using a theoretical framework called statistical teleodynamics, which is a
conceptual synthesis of statistical thermodynamics and potential game theory,
we predict that all highly connected layers of deep neural networks have a
universal microstructure of connection strengths that is distributed
lognormally ($LN({\mu}, {\sigma})$). Furthermore, under ideal conditions, the
theory predicts that ${\mu}$ and ${\sigma}$ are the same for all layers in all
networks. This is shown to be the result of an arbitrage equilibrium where all
connections compete and contribute the same effective utility towards the
minimization of the overall loss function. These surprising predictions are
shown to be supported by empirical data from six large-scale deep neural
networks in real life. We also discuss how these results can be exploited to
reduce the amount of data, time, and computational resources needed to train
large deep neural networks.

使用统计力学和潜在博弈理论的概念综合统计动力学的理论框架，我们提出了深度神经网络微观结构的一种新理论。该理论预测深度神经网络所有高度连接层的连接强度具有对数正态分布的普遍微观结构，在理想条件下所有网络的各层具有相同的均值和标准差。这些预测通过实际的大规模深度神经网络数据得到支持，并讨论了如何利用这些结果来减少训练大规模深度神经网络所需的数据、时间和计算资源。

Jaynes Machine: 深度神经网络的通用微结构

Jaynes Machine: The universal microstructure of deep neural networks

Structural equation models and Bayesian networks have been widely used to
analyze causal relations between continuous variables. In such frameworks,
linear acyclic models are typically used to model the data-generating process
of variables. Recently, it was shown that use of non-Gaussianity identifies the
full structure of a linear acyclic model, i.e., a causal ordering of variables
and their connection strengths, without using any prior knowledge on the
network structure, which is not the case with conventional methods. However,
existing estimation methods are based on iterative search algorithms and may
not converge to a correct solution in a finite number of steps. In this paper,
we propose a new direct method to estimate a causal ordering and connection
strengths based on non-Gaussianity.
In contrast to the previous methods, our algorithm requires no algorithmic
parameters and is guaranteed to converge to the right solution within a small
fixed number of steps if the data strictly follows the model.

本文介绍了一种基于非高斯性的新的、无需迭代算法参数即可估计因果排序和连接强度的方法，该方法能够在少量的步骤内保证求解正确，能够处理连续变量之间的因果关系。