In this paper, we provide a unified analysis of temporal difference learning
algorithms with linear function approximators by exploiting their connections
to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in
the control community to characterize the exact behaviors of the first and
second order moments of a large family of temporal difference learning
algorithms. For both the IID and Markov noise cases, we show that the evolution
of some augmented versions of the mean and covariance matrix of the TD
estimation error exactly follows the trajectory of a deterministic linear
time-invariant (LTI) dynamical system. Applying the well-known LTI system
theory, we obtain closed-form expressions for the mean and covariance matrix of
the TD estimation error at any time step. We provide a tight matrix spectral
radius condition to guarantee the convergence of the covariance matrix of the
TD estimation error, and perform a perturbation analysis to characterize the
dependence of the TD behaviors on learning rate. For the IID case, we provide
an exact formula characterizing how the mean and covariance matrix of the TD
estimation error converge to the steady state values. For the Markov case, we
use our formulas to explain how the behaviors of TD learning algorithms are
affected by learning rate and the underlying Markov chain. For both cases,
upper and lower bounds for the mean square TD error are provided. The mean
square TD error is shown to converge linearly to an exact limit.

本文通过探讨强化学习中时序差分学习算法与 Markov 跳线性系统（MJLS）之间的关联，为时序差分学习算法的收敛性和稳定性提供了理论支持，并在 IID 和 Markov 噪声的两种情形下提供了 TD 学习算法收敛和表现的精确性和上下界分析。

使用马尔科夫跳跃线性系统理论表征时序差分学习算法的确切行为

Characterizing the Exact Behaviors of Temporal Difference Learning  Algorithms Using Markov Jump Linear System Theory

Treatment effects can be estimated from observational data as the difference
in potential outcomes. In this paper, we address the challenge of estimating
the potential outcome when treatment-dose levels can vary continuously over
time. Further, the outcome variable may not be measured at a regular frequency.
Our proposed solution represents the treatment response curves using linear
time-invariant dynamical systems---this provides a flexible means for modeling
response over time to highly variable dose curves. Moreover, for multivariate
data, the proposed method: uncovers shared structure in treatment response and
the baseline across multiple markers; and, flexibly models challenging
correlation structure both across and within signals over time. For this, we
build upon the framework of multiple-output Gaussian Processes. On simulated
and a challenging clinical dataset, we show significant gains in accuracy over
state-of-the-art models.

本研究提出了一种利用线性时不变动力系统和高斯过程，能够弥补观察数据中剂量级别变化以及不定期观测的不足的方法，能够更加准确地进行多变量数据的建模和预测。