In Extreme Q-learning (XQL), Gumbel Regression is performed with an assumed
Gumbel distribution for the error distribution. This allows learning of the
value function without sampling out-of-distribution actions and has shown
excellent performance mainly in Offline RL. However, issues remained, including
the exponential term in the loss function causing instability and the potential
for an error distribution diverging from the Gumbel distribution. Therefore, we
propose Maclaurin Expanded Extreme Q-learning to enhance stability. In this
method, applying Maclaurin expansion to the loss function in XQL enhances
stability against large errors. It also allows adjusting the error distribution
assumption from normal to Gumbel based on the expansion order. Our method
significantly stabilizes learning in Online RL tasks from DM Control, where XQL
was previously unstable. Additionally, it improves performance in several
Offline RL tasks from D4RL, where XQL already showed excellent results.

通过应用 Maclaurin 展开来改进 Extreme Q-learning 方法，提高了稳定性，并允许根据展开阶数调整误差分布假设，从而显著稳定了在线强化学习任务，同时提高了离线强化学习任务的性能。

通过麦克劳林展开稳定极限 Q 学习

Stabilizing Extreme Q-learning by Maclaurin Expansion

Compression schemes have been extensively used in Federated Learning (FL) to
reduce the communication cost of distributed learning. While most approaches
rely on a bounded variance assumption of the noise produced by the compressor,
this paper investigates the use of compression and aggregation schemes that
produce a specific error distribution, e.g., Gaussian or Laplace, on the
aggregated data. We present and analyze different aggregation schemes based on
layered quantizers achieving exact error distribution. We provide different
methods to leverage the proposed compression schemes to obtain
compression-for-free in differential privacy applications. Our general
compression methods can recover and improve standard FL schemes with Gaussian
perturbations such as Langevin dynamics and randomized smoothing.

本文研究了在联邦学习中使用压缩和聚合方案来产生特定误差分布的方法，分析了基于分层量化实现准确误差分布的不同聚合方案，并提供了利用所提出的压缩方案在差分隐私应用中免费获得压缩的不同方法。我们的通用压缩方法可以恢复和改进具有高斯扰动的标准联邦学习方案，如 Langevin 动力学和随机平滑。