We present a novel statistical approach to incorporating uncertainty awareness in model-free distributional reinforcement learning involving quantile regression-based deep Q networks. The proposed algorithm, $\textit{Calibrated Evidential Quantile Regression in Deep Q Networks (CEQR-DQN)}$, aims to address key challenges associated with separately estimating aleatoric and epistemic uncertainty in stochastic environments. It combines deep evidential learning with quantile calibration based on principles of conformal inference to provide explicit, sample-free computations of $\textit{global}$ uncertainty as opposed to $\textit{local}$ estimates based on simple variance, overcoming limitations of traditional methods in computational and statistical efficiency and handling of out-of-distribution (OOD) observations. Tested on a suite of miniaturized Atari games (i.e., MinAtar), CEQR-DQN is shown to surpass similar existing frameworks in scores and learning speed. Its ability to rigorously evaluate uncertainty improves exploration strategies and can serve as a blueprint for other algorithms requiring uncertainty awareness.

我们提出了一种新颖的统计方法，用于在无模型分布式强化学习中纳入不确定性感知，它涉及基于分位回归的深度 Q 网络。该算法称为 CEQR-DQN（Calibrated Evidential Quantile Regression in Deep Q Networks），旨在解决在随机环境中分别估计偶然性和认识性不确定性所面临的关键挑战。它将深度证据学习与基于符合推理原则的分位校准相结合，提供明确的、无需样本的全局不确定性计算，而不是基于简单差异的局部估计，从而克服了传统方法在计算和统计效率以及处理超出分布范围观察的局限性。用于一套小型 Atari 游戏（即 MinAtar）的测试中，CEQR-DQN 在得分和学习速度方面超越了类似的现有框架。它对严格评估不确定性的能力改进了探索策略，并可作为其他需要不确定性感知的算法的蓝图。

苏格拉底的怀疑回声：在校准的证据强化学习中拥抱不确定性