State-of-the-art neural networks require extreme computational power to
train. It is therefore natural to wonder whether they are optimally trained.
Here we apply a recent advancement in stochastic thermodynamics which allows
bounding the speed at which one can go from the initial weight distribution to
the final distribution of the fully trained network, based on the ratio of
their Wasserstein-2 distance and the entropy production rate of the dynamical
process connecting them. Considering both gradient-flow and Langevin training
dynamics, we provide analytical expressions for these speed limits for linear
and linearizable neural networks e.g. Neural Tangent Kernel (NTK). Remarkably,
given some plausible scaling assumptions on the NTK spectra and spectral
decomposition of the labels -- learning is optimal in a scaling sense. Our
results are consistent with small-scale experiments with Convolutional Neural
Networks (CNNs) and Fully Connected Neural networks (FCNs) on CIFAR-10, showing
a short highly non-optimal regime followed by a longer optimal regime.

神经网络训练需要极高的计算能力，本文应用了随机热力学近期的进展来界定训练网络从初始权重分布转化为完全训练好的网络的速度上限，针对线性和可线性的神经网络提供了这些速度上限的解析表达式，发现在一些合理的缩放假设下，学习是在缩放意义上最优的，与卷积神经网络和全连接神经网络的实验结果相一致。

深度学习的速度限制

Speed Limits for Deep Learning

Reinforcement Learning (RL) models have continually evolved to navigate the
exploration - exploitation trade-off in uncertain Markov Decision Processes
(MDPs). In this study, I leverage the principles of stochastic thermodynamics
and system dynamics to explore reward shaping via diffusion processes. This
provides an elegant framework as a way to think about exploration-exploitation
trade-off. This article sheds light on relationships between information
entropy, stochastic system dynamics, and their influences on entropy
production. This exploration allows us to construct a dual-pronged framework
that can be interpreted as either a maximum entropy program for deriving
efficient policies or a modified cost optimization program accounting for
informational costs and benefits. This work presents a novel perspective on the
physical nature of information and its implications for online learning in
MDPs, consequently providing a better understanding of information-oriented
formulations in RL.

本研究利用随机热力学和系统动力学的原理，探索通过扩散过程进行奖励塑造的方法，为探索 - 开发权衡提供了一个优雅的框架，并揭示了信息熵、随机系统动力学之间的关系及其对熵产生的影响，从而构建了一个双重框架，可作为派生有效策略的最大熵程序或计算信息成本和利益的修改成本优化程序的解释，为 RL 中的信息导向公式提供了新的视角。

强化学习中的扩散过程奖励塑造

Reward Shaping via Diffusion Process in Reinforcement Learning

Stochastic thermodynamics as reviewed here systematically provides a
framework for extending the notions of classical thermodynamics like work, heat
and entropy production to the level of individual trajectories of well-defined
non-equilibrium ensembles. It applies whenever a non-equilibrium process is
still coupled to one (or several) heat bath(s) of constant temperature.
Paradigmatic systems are single colloidal particles in time-dependent laser
traps, polymers in external flow, enzymes and molecular motors in single
molecule assays, small biochemical networks and thermoelectric devices
involving single electron transport. For such systems, a first-law like energy
balance can be identified along fluctuating trajectories. Various integral and
detailed fluctuation theorems, which are derived here in a unifying approach
from one master theorem, constrain the probability distributions for work, heat
and entropy production depending on the nature of the system and the choice of
non-equilibrium conditions. For non-equilibrium steady states, particularly
strong results hold like a generalized fluctuation-dissipation theorem
involving entropy production. Ramifications and applications of these concepts
include optimal driving between specified states in finite time, the role of
measurement-based feedback processes and the relation between dissipation and
irreversibility. Efficiency and, in particular, efficiency at maximum power,
can be discussed systematically beyond the linear response regime for two
classes of molecular machines, isothermal ones like molecular motors, and heat
engines like thermoelectric devices, using a common framework based on a cycle
decomposition of entropy production.

随机热力学是将经典热力学的概念（如功、热以及熵的产生）扩展到非平衡集合的个体轨迹的框架，主要应用于单胶体颗粒、聚合物、酶和分子电机、小型生化网络以及热电器件等系统中，各种波动定理限制着功、热和熵产生的概率分布，并可以通过一个主定理来规定。