Reinforcement learning has recently gained unprecedented popularity, yet it
still grapples with sample inefficiency. Addressing this challenge, federated
reinforcement learning (FedRL) has emerged, wherein agents collaboratively
learn a single policy by aggregating local estimations. However, this
aggregation step incurs significant communication costs. In this paper, we
propose CompFedRL, a communication-efficient FedRL approach incorporating both
\textit{periodic aggregation} and (direct/error-feedback) compression
mechanisms. Specifically, we consider compressed federated $Q$-learning with a
generative model setup, where a central server learns an optimal $Q$-function
by periodically aggregating compressed $Q$-estimates from local agents. For the
first time, we characterize the impact of these two mechanisms (which have
remained elusive) by providing a finite-time analysis of our algorithm,
demonstrating strong convergence behaviors when utilizing either direct or
error-feedback compression. Our bounds indicate improved solution accuracy
concerning the number of agents and other federated hyperparameters while
simultaneously reducing communication costs. To corroborate our theory, we also
conduct in-depth numerical experiments to verify our findings, considering
Top-$K$ and Sparsified-$K$ sparsification operators.

提出了一种通信高效的联合强化学习方法，该方法利用周期性聚合和压缩机制将本地代理的压缩估计聚合到一个中央服务器上，首次对这两种机制进行了有限时间分析，并在实验中验证了算法的收敛性行为。

带有生成模型的压缩联邦强化学习

Compressed Federated Reinforcement Learning with a Generative Model

Compressive learning is an emerging approach to drastically reduce the memory
footprint of large-scale learning, by first summarizing a large dataset into a
low-dimensional sketch vector, and then decoding from this sketch the latent
information needed for learning. In light of recent progress on information
preservation guarantees for sketches based on random features, a major
objective is to design easy-to-tune algorithms (called decoders) to robustly
and efficiently extract this information. To address the underlying non-convex
optimization problems, various heuristics have been proposed. In the case of
compressive clustering, the standard heuristic is CL-OMPR, a variant of sliding
Frank-Wolfe. Yet, CL-OMPR is hard to tune, and the examination of its
robustness was overlooked. In this work, we undertake a scrutinized examination
of CL-OMPR to circumvent its limitations. In particular, we show how this
algorithm can fail to recover the clusters even in advantageous scenarios. To
gain insight, we show how the deficiencies of this algorithm can be attributed
to optimization difficulties related to the structure of a correlation function
appearing at core steps of the algorithm. To address these limitations, we
propose an alternative decoder offering substantial improvements over CL-OMPR.
Its design is notably inspired from the mean shift algorithm, a classic
approach to detect the local maxima of kernel density estimators. The proposed
algorithm can extract clustering information from a sketch of the MNIST dataset
that is 10 times smaller than previously.

压缩学习是一种新兴方法，通过首先将大规模数据集总结为低维度的草图向量，然后从这个草图中解码出学习所需的潜在信息，从而极大地减少大规模学习的内存占用。我们的研究旨在设计易于调优的算法（称为解码器），以稳健高效地提取这些信息，然后通过分析压缩聚类中的 CL-OMPR 算法的局限性，提出了一种改进方案，大大提高了信息提取的准确性和效率。

草图与移动：一种强健的聚类压缩解码器

Sketch and shift: a robust decoder for compressive clustering

Learning parameters from voluminous data can be prohibitive in terms of
memory and computational requirements. We propose a "compressive learning"
framework where we estimate model parameters from a sketch of the training
data. This sketch is a collection of generalized moments of the underlying
probability distribution of the data. It can be computed in a single pass on
the training set, and is easily computable on streams or distributed datasets.
The proposed framework shares similarities with compressive sensing, which aims
at drastically reducing the dimension of high-dimensional signals while
preserving the ability to reconstruct them. To perform the estimation task, we
derive an iterative algorithm analogous to sparse reconstruction algorithms in
the context of linear inverse problems. We exemplify our framework with the
compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics
on the choice of the sketching procedure and theoretical guarantees of
reconstruction. We experimentally show on synthetic data that the proposed
algorithm yields results comparable to the classical Expectation-Maximization
(EM) technique while requiring significantly less memory and fewer computations
when the number of database elements is large. We further demonstrate the
potential of the approach on real large-scale data (over 10 8 training samples)
for the task of model-based speaker verification. Finally, we draw some
connections between the proposed framework and approximate Hilbert space
embedding of probability distributions using random features. We show that the
proposed sketching operator can be seen as an innovative method to design
translation-invariant kernels adapted to the analysis of GMMs. We also use this
theoretical framework to derive information preservation guarantees, in the
spirit of infinite-dimensional compressive sensing.

本文提出了一种压缩学习框架，通过数据的限制集来估计模型参数，并演示了该方法在高斯混合模型和说话人验证任务中的成功应用，以及它与随机特征的近似希尔伯特空间嵌入方法之间的联系。