We have formulated generative machine learning problems as the time evolution
of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic
process. Then, we have studied the thermodynamic exchange between the model's
parameters, denoted as $\Theta$, and the model's generated samples, denoted as
$X$. We demonstrate that the training dataset and the action of the Stochastic
Gradient Descent (SGD) optimizer serve as a work source that governs the time
evolution of these two subsystems. Our findings reveal that the model learns
through the dissipation of heat during the generation of samples $X$, leading
to an increase in the entropy of the model's parameters, $\Theta$. Thus, the
parameter subsystem acts as a heat reservoir, effectively storing the learned
information. Furthermore, the role of the model's parameters as a heat
reservoir provides valuable thermodynamic insights into the generalization
power of over-parameterized models. This approach offers an unambiguous
framework for computing information-theoretic quantities within deterministic
neural networks by establishing connections with thermodynamic variables. To
illustrate the utility of this framework, we introduce two
information-theoretic metrics: Memorized-information (M-info) and
Learned-information (L-info), which trace the dynamic flow of information
during the learning process of PPMs.

我们通过建立与热力学变量的联系，将生成式机器学习问题形式化为参数概率模型（PPM）的时间演化，研究了模型参数（θ）和生成样本（X）之间的热力学交换。我们发现模型通过样本生成过程中的热量耗散来学习，导致模型参数熵增加，进而作为热库保存学到的信息。此方法提供了一个计算神经网络内信息论量的明确框架，并引入了两个信息论度量指标：记忆化信息（M-info）和学习化信息（L-info），用于追踪 PPM 学习过程中的信息动态流动。

学习生成参数概率模型的随机热力学

Stochastic Thermodynamics of Learning Generative Parametric  Probabilistic Models

Many real-world problems are inherently multimodal, from the communicative
modalities humans use to express social and emotional states to the force,
proprioception, and visual sensors ubiquitous on robots. While there has been
an explosion of interest in multimodal representation learning, these methods
are still largely focused on a small set of modalities, primarily in the
language, vision, and audio space. In order to accelerate generalization
towards diverse and understudied modalities, this paper studies efficient
representation learning for high-modality scenarios. Since adding new models
for every new modality or task becomes prohibitively expensive, a critical
technical challenge is heterogeneity quantification: how can we measure which
modalities encode similar information and interactions in order to permit
parameter sharing with previous modalities? We propose two new
information-theoretic metrics for heterogeneity quantification: (1) modality
heterogeneity studies how similar 2 modalities $\{X_1,X_2\}$ are by measuring
how much information can be transferred from $X_1$ to $X_2$, while (2)
interaction heterogeneity studies how similarly pairs of modalities
$\{X_1,X_2\}, \{X_3,X_4\}$ interact by measuring how much interaction
information can be transferred from $\{X_1,X_2\}$ to $\{X_3,X_4\}$. We show the
importance of these proposed metrics in high-modality scenarios as a way to
automatically prioritize the fusion of modalities that contain unique
information or interactions. The result is a single model, HighMMT, that scales
up to $10$ modalities and $15$ tasks from $5$ different research areas. Not
only does HighMMT outperform prior methods on the tradeoff between performance
and efficiency, it also demonstrates a crucial scaling behavior: performance
continues to improve with each modality added, and transfers to entirely new
modalities and tasks during fine-tuning.

本文提出了两种信息理论的度量方法，用于量化多模态学习中异构性，并提出了一种可扩展到多个领域的高模态模型 HighMMT，其性能优于以前的方法，并表现出重要的缩放行为。

HighMMT: 量化多模态和交互异质性实现高维表示学习

HighMMT: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

In this work, we attempt to explain the prediction of any black-box
classifier from an information-theoretic perspective. For each input feature,
we compare the classifier outputs with and without that feature using two
information-theoretic metrics. Accordingly, we obtain two attribution maps--an
information gain (IG) map and a point-wise mutual information (PMI) map. IG map
provides a class-independent answer to "How informative is each pixel?", and
PMI map offers a class-specific explanation of "How much does each pixel
support a specific class?" Compared to existing methods, our method improves
the correctness of the attribution maps in terms of a quantitative metric. We
also provide a detailed analysis of an ImageNet classifier using the proposed
method, and the code is available online.

该研究从信息熵角度试图解释任意黑匣子分类器的预测结果，比现有方法提高了量化评价的正确性，通过提供两个信息熵评估指标（信息增益图和逐点互信息图）来分别提供类别无关和类别特定的结果。作者还对 ImageNet 分类器进行了详细分析，并提供了在线代码。