MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that
extends the popular single-agent method, DDPG, to multi-agent scenarios.
Importantly, DDPG is an algorithm designed for continuous action spaces, where
the gradient of the state-action value function exists. For this algorithm to
work in discrete action spaces, discrete gradient estimation must be performed.
For MADDPG, the Gumbel-Softmax (GS) estimator is used -- a reparameterisation
which relaxes a discrete distribution into a similar continuous one. This
method, however, is statistically biased, and a recent MARL benchmarking paper
suggests that this bias makes MADDPG perform poorly in grid-world situations,
where the action space is discrete. Fortunately, many alternatives to the GS
exist, boasting a wide range of properties. This paper explores several of
these alternatives and integrates them into MADDPG for discrete grid-world
scenarios. The corresponding impact on various performance metrics is then
measured and analysed. It is found that one of the proposed estimators performs
significantly better than the original GS in several tasks, achieving up to 55%
higher returns, along with faster convergence.

本文探讨了在离散动作空间的场景下，使用多种代替 Gumbel-Softmax 估计器的方法来扩展 MADDPG 算法，并对各种性能指标进行了测量和分析，结果表明，在几项任务中，其中一种提出的估计方法比原始的 Gumbel-Softmax 在返回率上表现显著更好，同时收敛更快。

重温使用 Gumbel-Softmax 的 MADDPG 算法

Revisiting the Gumbel-Softmax in MADDPG

Gradient estimation in models with discrete latent variables is a challenging
problem, because the simplest unbiased estimators tend to have high variance.
To counteract this, modern estimators either introduce bias, rely on multiple
function evaluations, or use learned, input-dependent baselines. Thus, there is
a need for estimators that require minimal tuning, are computationally cheap,
and have low mean squared error. In this paper, we show that the variance of
the straight-through variant of the popular Gumbel-Softmax estimator can be
reduced through Rao-Blackwellization without increasing the number of function
evaluations. This provably reduces the mean squared error. We empirically
demonstrate that this leads to variance reduction, faster convergence, and
generally improved performance in two unsupervised latent variable models.

本研究提出了一种对 Gumbel-Softmax estimator 进行 Rao-Blackwellization 的方法，可在不增加函数评估数量的情况下减少方差，从而降低均方误差，并在两种无监督潜变量模型中得到了实证验证。

Rao-Blackwell 算法优化直通 Gumbel-Softmax 梯度估计器

Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient  Estimator

Recent work in network quantization produced state-of-the-art results using
mixed precision quantization. An imperative requirement for many efficient edge
device hardware implementations is that their quantizers are uniform and with
power-of-two thresholds. In this work, we introduce the Hardware Friendly Mixed
Precision Quantization Block (HMQ) in order to meet this requirement. The HMQ
is a mixed precision quantization block that repurposes the Gumbel-Softmax
estimator into a smooth estimator of a pair of quantization parameters, namely,
bit-width and threshold. HMQs use this to search over a finite space of
quantization schemes. Empirically, we apply HMQs to quantize classification
models trained on CIFAR10 and ImageNet. For ImageNet, we quantize four
different architectures and show that, in spite of the added restrictions to
our quantization scheme, we achieve competitive and, in some cases,
state-of-the-art results.

本文提出了 HMQ，一个基于 Gumbel-Softmax 估计器的混合精度量化块，搜索有限的量化方案空间，以覆盖在边缘设备硬件实现中需要的均匀和具有 2 的幂级别阈值的量化器需求，并在 CIFAR10 和 ImageNet 训练的分类模型上应用 HMQ，证明在一些情况下，虽然添加了限制，但也能够实现竞争性和最先进的结果。