Exploration in reinforcement learning is a challenging problem: in the worst
case, the agent must search for high-reward states that could be hidden
anywhere in the state space. Can we define a more tractable class of RL
problems, where the agent is provided with examples of successful outcomes? In
this problem setting, the reward function can be obtained automatically by
training a classifier to categorize states as successful or not. If trained
properly, such a classifier can provide a well-shaped objective landscape that
both promotes progress toward good states and provides a calibrated exploration
bonus. In this work, we show that an uncertainty aware classifier can solve
challenging reinforcement learning problems by both encouraging exploration and
provided directed guidance towards positive outcomes. We propose a novel
mechanism for obtaining these calibrated, uncertainty-aware classifiers based
on an amortized technique for computing the normalized maximum likelihood (NML)
distribution. To make this tractable, we propose a novel method for computing
the NML distribution by using meta-learning. We show that the resulting
algorithm has a number of intriguing connections to both count-based
exploration methods and prior algorithms for learning reward functions, while
also providing more effective guidance towards the goal. We demonstrate that
our algorithm solves a number of challenging navigation and robotic
manipulation tasks which prove difficult or impossible for prior methods.

采用基于归一化最大似然分布的元学习技术来获得不确定性感知的分类器对于鼓励探索和提供正向结果的引导至关重要。本文提出了一种计算归一化最大似然分布的新方法，并展示了该算法在导航和机器人操作任务中具有实际应用的有效性。