Learning about many things can provide numerous benefits to a reinforcement
learning system. For example, learning many auxiliary value functions, in
addition to optimizing the environmental reward, appears to improve both
exploration and representation learning. The question we tackle in this paper
is how to sculpt the stream of experience---how to adapt the learning system's
behavior---to optimize the learning of a collection of value functions. A
simple answer is to compute an intrinsic reward based on the statistics of each
auxiliary learner, and use reinforcement learning to maximize that intrinsic
reward. Unfortunately, implementing this simple idea has proven difficult, and
thus has been the focus of decades of study. It remains unclear which of the
many possible measures of learning would work well in a parallel learning
setting where environmental reward is extremely sparse or absent. In this
paper, we investigate and compare different intrinsic reward mechanisms in a
new bandit-like parallel-learning testbed. We discuss the interaction between
reward and prediction learners and highlight the importance of introspective
prediction learners: those that increase their rate of learning when progress
is possible, and decrease when it is not. We provide a comprehensive empirical
comparison of 14 different rewards, including well-known ideas from
reinforcement learning and active learning. Our results highlight a simple but
seemingly powerful principle: intrinsic rewards based on the amount of learning
can generate useful behavior, if each individual learner is introspective.

本文通过在一个类似于赌博机的并行学习测试平台中比较 14 个不同的回报机制，探索并比较不同的内在回报机制，重点突出了奖励和预测学习器之间的交互作用和内省预测学习器的重要性。结果表明，基于学习量的内在奖励可以生成有用的行为，如果每个学习器是内省的。