We develop a thermodynamic theory for machine learning (ML) systems. Similar
to physical thermodynamic systems which are characterized by energy and
entropy, ML systems possess these characteristics as well. This comparison
inspire us to integrate the concept of temperature into ML systems grounded in
the fundamental principles of thermodynamics, and establish a basic
thermodynamic framework for machine learning systems with non-Boltzmann
distributions. We introduce the concept of states within a ML system, identify
two typical types of state, and interpret model training and refresh as a
process of state phase transition. We consider that the initial potential
energy of a ML system is described by the model's loss functions, and the
energy adheres to the principle of minimum potential energy. For a variety of
energy forms and parameter initialization methods, we derive the temperature of
systems during the phase transition both analytically and asymptotically,
highlighting temperature as a vital indicator of system data distribution and
ML training complexity. Moreover, we perceive deep neural networks as complex
heat engines with both global temperature and local temperatures in each layer.
The concept of work efficiency is introduced within neural networks, which
mainly depends on the neural activation functions. We then classify neural
networks based on their work efficiency, and describe neural networks as two
types of heat engines.

开发了一种热力学理论用于机器学习系统，与物理热力学系统相似的是，机器学习系统也具有能量和熵的特征。我们引入了温度的概念，并建立了一个基本的热力学框架来处理具有非 Boltzmann 分布的机器学习系统。我们将机器学习系统看作具有不同状态的系统，并将模型训练和更新解释为状态相变的过程。我们将机器学习系统的初始潜在能量描述为模型的损失函数，并遵循最小潜在能量原则。我们推导了系统在相变过程中的温度，突出温度作为系统数据分布和机器学习训练复杂性的重要指标。此外，我们将深度神经网络视为具有全局温度和每层局部温度的复杂热能引擎，并介绍了神经网络的工作效率概念，主要取决于神经激活函数。然后，我们根据工作效率对神经网络进行分类，并将神经网络描述为两种类型的热能引擎。