Exploration is essential for reinforcement learning (RL). To face the
challenges of exploration, we consider a reward-free RL framework that
completely separates exploration from exploitation and brings new challenges
for exploration algorithms. In the exploration phase, the agent learns an
exploratory policy by interacting with a reward-free environment and collects a
dataset of transitions by executing the policy. In the planning phase, the
agent computes a good policy for any reward function based on the dataset
without further interacting with the environment. This framework is suitable
for the meta RL setting where there are many reward functions of interest. In
the exploration phase, we propose to maximize the Renyi entropy over the
state-action space and justify this objective theoretically. The success of
using Renyi entropy as the objective results from its encouragement to explore
the hard-to-reach state-actions. We further deduce a policy gradient
formulation for this objective and design a practical exploration algorithm
that can deal with complex environments. In the planning phase, we solve for
good policies given arbitrary reward functions using a batch RL algorithm.
Empirically, we show that our exploration algorithm is effective and sample
efficient, and results in superior policies for arbitrary reward functions in
the planning phase.

通过最大化 Renyi 熵的方法，提出了一种适用于元 RL 的无奖励强化学习框架，该框架有效地解决了探索和利用分离的问题，并设计了相应的强化学习算法 (batch RL algorithm) 以便在规划阶段中能更好地处理任意奖励函数。