We study the dynamics of a continuous-time model of the Stochastic Gradient
Descent (SGD) for the least-square problem. Indeed, pursuing the work of Li et
al. (2019), we analyze Stochastic Differential Equations (SDEs) that model SGD
either in the case of the training loss (finite samples) or the population one
(online setting). A key qualitative feature of the dynamics is the existence of
a perfect interpolator of the data, irrespective of the sample size. In both
scenarios, we provide precise, non-asymptotic rates of convergence to the
(possibly degenerate) stationary distribution. Additionally, we describe this
asymptotic distribution, offering estimates of its mean, deviations from it,
and a proof of the emergence of heavy-tails related to the step-size magnitude.
Numerical simulations supporting our findings are also presented.

我们研究了最小二乘问题的连续时间随机梯度下降（SGD）模型的动力学。我们通过分析随机微分方程 (SDE)，在训练损失（有限样本）或总体损失（在线设置）的情况下建模 SGD 来追求 Li 等人 (2019) 的研究成果。该动力学的一个关键特征是无论样本大小如何，都存在与数据完美插值器。在这两种情况下，我们提供了收敛到（可能退化的）稳态分布的精确非渐近速率。此外，我们描述了渐近分布，给出了其均值、与之偏差的估计，并证明了与步长大小有关的重尾现象的出现。我们还呈现了支持我们发现的数值模拟结果。

最小二乘随机梯度下降的随机微分方程模型

Stochastic Differential Equations models for Least-Squares Stochastic  Gradient Descent

Switching dynamical systems are an expressive model class for the analysis of
time-series data. As in many fields within the natural and engineering
sciences, the systems under study typically evolve continuously in time, it is
natural to consider continuous-time model formulations consisting of switching
stochastic differential equations governed by an underlying Markov jump
process. Inference in these types of models is however notoriously difficult,
and tractable computational schemes are rare. In this work, we propose a novel
inference algorithm utilizing a Markov Chain Monte Carlo approach. The
presented Gibbs sampler allows to efficiently obtain samples from the exact
continuous-time posterior processes. Our framework naturally enables Bayesian
parameter estimation, and we also include an estimate for the diffusion
covariance, which is oftentimes assumed fixed in stochastic differential
equation models. We evaluate our framework under the modeling assumption and
compare it against an existing variational inference approach.

该文提出了一种基于马尔科夫链蒙特卡罗方法的新型推断算法，用于在连续时间模型中获得精确的后验过程的样本，实现了贝叶斯参数估计，并估计了扩散协方差。

连续时间切换动态系统的马尔可夫链蒙特卡罗方法

Markov Chain Monte Carlo for Continuous-Time Switching Dynamical Systems

Motivated by the celebrated discrete-time model of nervous activity outlined
by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the
McCulloch-Pitts network (MPN), for sequence learning in spiking neural
networks. Our model has a local learning rule, such that the synaptic weight
updates depend only on the information directly accessible by the synapse. By
exploiting asymmetry in the connections between binary neurons, we show that
MPN can be trained to robustly memorize multiple spatiotemporal patterns of
binary vectors, generalizing the ability of the symmetric Hopfield network to
memorize static spatial patterns. In addition, we demonstrate that the model
can efficiently learn sequences of binary pictures as well as generative models
for experimental neural spike-train data. Our learning rule is consistent with
spike-timing-dependent plasticity (STDP), thus providing a theoretical ground
for the systematic design of biologically inspired networks with large and
robust long-range sequence storage capacity.

提出了一种新的持续时间模型 ——McCulloch-Pitts 网络 (MPN)， 用于序列学习，在棘突神经网络中，可以利用突触之间的不对称连接进行训练，能够稳健地记忆多个时空模式、二进制图片序列和实验神经脉冲数据的产生模型，并具有生物启发式的设计理念。