Exploration strategies in continuous action space are often heuristic due to
the infinite actions, and these kinds of methods cannot derive a general
conclusion. In prior work, it has been shown that policy-based exploration is
beneficial for continuous action space in deterministic policy reinforcement
learning(DPRL). However, policy-based exploration in DPRL has two prominent
issues: aimless exploration and policy divergence, and the policy gradient for
exploration is only sometimes helpful due to inaccurate estimation. Based on
the double-Q function framework, we introduce a novel exploration strategy to
mitigate these issues, separate from the policy gradient. We first propose the
greedy Q softmax update schema for Q value update. The expected Q value is
derived by weighted summing the conservative Q value over actions, and the
weight is the corresponding greedy Q value. Greedy Q takes the maximum value of
the two Q functions, and conservative Q takes the minimum value of the two
different Q functions. For practicality, this theoretical basis is then
extended to allow us to combine action exploration with the Q value update,
except for the premise that we have a surrogate policy that behaves like this
exploration policy. In practice, we construct such an exploration policy with a
few sampled actions, and to meet the premise, we learn such a surrogate policy
by minimizing the KL divergence between the target policy and the exploration
policy constructed by the conservative Q. We evaluate our method on the Mujoco
benchmark and demonstrate superior performance compared to previous
state-of-the-art methods across various environments, particularly in the most
complex Humanoid environment.

基于双 Q 函数框架，引入一种新的探索策略来解决连续动作空间中政策梯度法探索的问题，通过使用贪婪 Q 值和保守 Q 值的加权和来更新 Q 值，将行动探索与 Q 值更新相结合，并在 Mujoco 基准测试中展示了优越的性能。