Communication is crucial in multi-agent reinforcement learning when agents
are not able to observe the full state of the environment. The most common
approach to allow learned communication between agents is the use of a
differentiable communication channel that allows gradients to flow between
agents as a form of feedback. However, this is challenging when we want to use
discrete messages to reduce the message size, since gradients cannot flow
through a discrete communication channel. Previous work proposed methods to
deal with this problem. However, these methods are tested in different
communication learning architectures and environments, making it hard to
compare them. In this paper, we compare several state-of-the-art discretization
methods as well as a novel approach. We do this comparison in the context of
communication learning using gradients from other agents and perform tests on
several environments. In addition, we present COMA-DIAL, a communication
learning approach based on DIAL and COMA extended with learning rate scaling
and adapted exploration. Using COMA-DIAL allows us to perform experiments on
more complex environments. Our results show that the novel ST-DRU method,
proposed in this paper, achieves the best results out of all discretization
methods across the different environments. It achieves the best or close to the
best performance in each of the experiments and is the only method that does
not fail on any of the tested environments.

在多智能体强化学习中，当智能体无法观察到完整的环境状态时，沟通是至关重要的。借助可微分的通信渠道允许梯度在智能体之间流动作为一种反馈的最常见方法；然而，当我们想要使用离散消息来减小消息大小时，这种方法面临挑战，因为梯度无法通过离散通信渠道传递。本文比较了几种最先进的离散化方法和一种新方法，并将比较应用于基于梯度的沟通学习，并在多个环境中进行了测试。此外，本文还提出了一种基于 DIAL 和 COMA 的沟通学习方法 COMA-DIAL，该方法通过学习率缩放和适应性探索进行了扩展，使我们能够在更复杂的环境中进行实验。实验结果表明，本文提出的新方法 ST-DRU 在各种环境中都取得了最佳结果，每个实验中的表现最好或接近最佳，并且是唯一一个在任何测试环境中都没有失败的方法。