Learning to reach goal states and learning diverse skills through mutual
information (MI) maximization have been proposed as principled frameworks for
self-supervised reinforcement learning, allowing agents to acquire broadly
applicable multitask policies with minimal reward engineering. Starting from a
simple observation that the standard goal-conditioned RL (GCRL) is encapsulated
by the optimization objective of variational empowerment, we discuss how GCRL
and MI-based RL can be generalized into a single family of methods, which we
name variational GCRL (VGCRL), interpreting variational MI maximization, or
variational empowerment, as representation learning methods that acquire
functionally-aware state representations for goal reaching. This novel
perspective allows us to: (1) derive simple but unexplored variants of GCRL to
study how adding small representation capacity can already expand its
capabilities; (2) investigate how discriminator function capacity and
smoothness determine the quality of discovered skills, or latent goals, through
modifying latent dimensionality and applying spectral normalization; (3) adapt
techniques such as hindsight experience replay (HER) from GCRL to MI-based RL;
and lastly, (4) propose a novel evaluation metric, named latent goal reaching
(LGR), for comparing empowerment algorithms with different choices of latent
dimensionality and discriminator parameterization. Through principled
mathematical derivations and careful experimental studies, our work lays a
novel foundation from which to evaluate, analyze, and develop representation
learning techniques in goal-based RL.

通过代表愿景达成的功能感知状态表示进行的变分互信息最大化，能够为达到愿景状态的广泛应用的多任务策略的自监督强化学习提供框架和方法，同时还提出了广义 GCRL 和 MI-Based RL 的统一方法，即 VGCRL，并结合方法的容量和光滑性分析了能力扩展，以及其与不同线性变换结构的潜在目标发现算法的比较评价指标，即 LGR。