We consider a distributed setup for reinforcement learning, where each agent
has a copy of the same Markov Decision Process but transitions are sampled from
the corresponding Markov chain independently by each agent. We show that in
this setting, we can achieve a linear speedup for TD($\lambda$), a family of
popular methods for policy evaluation, in the sense that $N$ agents can
evaluate a policy $N$ times faster provided the target accuracy is small
enough. Notably, this speedup is achieved by ``one shot averaging,'' a
procedure where the agents run TD($\lambda$) with Markov sampling independently
and only average their results after the final step. This significantly reduces
the amount of communication required to achieve a linear speedup relative to
previous work.

我们考虑一种分布式设置的强化学习，其中每个智能体都有相同的马尔可夫决策过程的副本，但是转移矩阵由每个智能体独立进行采样。我们表明在这个设置中，我们可以通过 ' 一次性平均 ' 的过程，使得 N 个智能体对策略进行 N 倍的速度加速，前提是目标准确性足够小。这种加速是相对于先前的工作来说通信所需量大大减少的线性加速方法。

分布式 TD ($λ$) 的单次平均化方法在马尔可夫采样下应用

One-Shot Averaging for Distributed TD($λ$) Under Markov Sampling

This paper is devoted to studying the semi-supervised sparse statistical
inference in a distributed setup. An efficient multi-round distributed debiased
estimator, which integrates both labeled and unlabelled data, is developed. We
will show that the additional unlabeled data helps to improve the statistical
rate of each round of iteration. Our approach offers tailored debiasing methods
for $M$-estimation and generalized linear model according to the specific form
of the loss function. Our method also applies to a non-smooth loss like
absolute deviation loss. Furthermore, our algorithm is computationally
efficient since it requires only one estimation of a high-dimensional inverse
covariance matrix. We demonstrate the effectiveness of our method by presenting
simulation studies and real data applications that highlight the benefits of
incorporating unlabeled data.

本研究旨在研究半监督稀疏统计推断在分布式环境中的应用，提出了一种高效的多轮分布式去偏估计方法，有效地整合了有标记和无标记的数据，并应用于 M-estimation 和广义线性模型等不同损失函数形式。通过模拟研究和真实数据应用，证明了这种方法整合了未标记数据的好处。