Alleviating overestimation bias is a critical challenge for deep
reinforcement learning to achieve successful performance on more complex tasks
or offline datasets containing out-of-distribution data. In order to overcome
overestimation bias, ensemble methods for Q-learning have been investigated to
exploit the diversity of multiple Q-functions. Since network initialization has
been the predominant approach to promote diversity in Q-functions,
heuristically designed diversity injection methods have been studied in the
literature. However, previous studies have not attempted to approach guaranteed
independence over an ensemble from a theoretical perspective. By introducing a
novel regularization loss for Q-ensemble independence based on random matrix
theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR)
for reinforcement learning. Specifically, we modify the intractable hypothesis
testing criterion for the Q-ensemble independence into a tractable KL
divergence between the spectral distribution of the Q-ensemble and the target
Wigner's semicircle distribution. We implement SPQR in several online and
offline ensemble Q-learning algorithms. In the experiments, SPQR outperforms
the baseline algorithms in both online and offline RL benchmarks.

为了克服过度估计偏差，在深度强化学习中提出了一种新的正则化损失函数，基于随机矩阵理论的销钉 Wishart Q 集合独立正则化（SPQR），以增强多个 Q 函数的多样性，通过实现 SPQR 在在线和离线集合 Q 学习算法中进行实验证明，SPQR 在在线和离线强化学习基准测试中表现出色。