In complex environments with large discrete action spaces, effective decision-making is critical in reinforcement learning (RL). Despite the widespread use of value-based RL approaches like Q-learning, they come with a computational burden, necessitating the maximization of a value function over all actions in each iteration. This burden becomes particularly challenging when addressing large-scale problems and using deep neural networks as function approximators. In this paper, we present stochastic value-based RL approaches which, in each iteration, as opposed to optimizing over the entire set of $n$ actions, only consider a variable stochastic set of a sublinear number of actions, possibly as small as $\mathcal{O}(\log(n))$. The presented stochastic value-based RL methods include, among others, Stochastic Q-learning, StochDQN, and StochDDQN, all of which integrate this stochastic approach for both value-function updates and action selection. The theoretical convergence of Stochastic Q-learning is established, while an analysis of stochastic maximization is provided. Moreover, through empirical validation, we illustrate that the various proposed approaches outperform the baseline methods across diverse environments, including different control problems, achieving near-optimal average returns in significantly reduced time.

在复杂环境中，使用深度神经网络作为函数近似器的大离散行为空间的强化学习问题中，传统的基于值的强化学习方法存在计算负担，本文提出了一种用于优化前 $n$ 个动作集合的变量随机子集的基于值的强化学习方法，并通过理论证明和实证验证表明，在不同控制问题上，这些方法在减少时间的同时仍能实现接近最优的平均回报。

大规模离散动作空间的随机 Q 学习