We have formulated generative machine learning problems as the time evolution
of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic
process. Then, we have studied the thermodynamic exchange between the model's
parameters, denoted as $\Theta$, and the model's generated samples, denoted as
$X$. We demonstrate that the training dataset and the action of the Stochastic
Gradient Descent (SGD) optimizer serve as a work source that governs the time
evolution of these two subsystems. Our findings reveal that the model learns
through the dissipation of heat during the generation of samples $X$, leading
to an increase in the entropy of the model's parameters, $\Theta$. Thus, the
parameter subsystem acts as a heat reservoir, effectively storing the learned
information. Furthermore, the role of the model's parameters as a heat
reservoir provides valuable thermodynamic insights into the generalization
power of over-parameterized models. This approach offers an unambiguous
framework for computing information-theoretic quantities within deterministic
neural networks by establishing connections with thermodynamic variables. To
illustrate the utility of this framework, we introduce two
information-theoretic metrics: Memorized-information (M-info) and
Learned-information (L-info), which trace the dynamic flow of information
during the learning process of PPMs.

我们通过建立与热力学变量的联系，将生成式机器学习问题形式化为参数概率模型（PPM）的时间演化，研究了模型参数（θ）和生成样本（X）之间的热力学交换。我们发现模型通过样本生成过程中的热量耗散来学习，导致模型参数熵增加，进而作为热库保存学到的信息。此方法提供了一个计算神经网络内信息论量的明确框架，并引入了两个信息论度量指标：记忆化信息（M-info）和学习化信息（L-info），用于追踪 PPM 学习过程中的信息动态流动。