Goal-Conditioned Reinforcement Learning (RL) problems often have access to
sparse rewards where the agent receives a reward signal only when it has
achieved the goal, making policy optimization a difficult problem. Several
works augment this sparse reward with a learned dense reward function, but this
can lead to sub-optimal policies if the reward is misaligned. Moreover, recent
works have demonstrated that effective shaping rewards for a particular problem
can depend on the underlying learning algorithm. This paper introduces a novel
way to encourage exploration called $f$-Policy Gradients, or $f$-PG. $f$-PG
minimizes the f-divergence between the agent's state visitation distribution
and the goal, which we show can lead to an optimal policy. We derive gradients
for various f-divergences to optimize this objective. Our learning paradigm
provides dense learning signals for exploration in sparse reward settings. We
further introduce an entropy-regularized policy optimization objective, that we
call $state$-MaxEnt RL (or $s$-MaxEnt RL) as a special case of our objective.
We show that several metric-based shaping rewards like L2 can be used with
$s$-MaxEnt RL, providing a common ground to study such metric-based shaping
rewards with efficient exploration. We find that $f$-PG has better performance
compared to standard policy gradient methods on a challenging gridworld as well
as the Point Maze and FetchReach environments. More information on our website
this https URL

此研究论文介绍了一种名为 $f$-PG 的新型鼓励探索方法，通过最小化智能体状态访问分布与目标之间的 f - 分歧来实现稀疏奖励环境下的优化策略，同时引入了熵正则化策略优化目标 $s$-MaxEnt RL 用于优化度量为 L2 的奖励，在多个环境中展示了与标准策略梯度方法相比更好的性能。