Offline reinforcement learning (RL) aims to learn an effective policy from a
pre-collected dataset. Most existing works are to develop sophisticated
learning algorithms, with less emphasis on improving the data collection
process. Moreover, it is even challenging to extend the single-task setting and
collect a task-agnostic dataset that allows an agent to perform multiple
downstream tasks. In this paper, we propose a Curiosity-driven Unsupervised
Data Collection (CUDC) method to expand feature space using adaptive temporal
distances for task-agnostic data collection and ultimately improve learning
efficiency and capabilities for multi-task offline RL. To achieve this, CUDC
estimates the probability of the k-step future states being reachable from the
current states, and adapts how many steps into the future that the dynamics
model should predict. With this adaptive reachability mechanism in place, the
feature representation can be diversified, and the agent can navigate itself to
collect higher-quality data with curiosity. Empirically, CUDC surpasses
existing unsupervised methods in efficiency and learning performance in various
downstream offline RL tasks of the DeepMind control suite.

该论文提出了一种基于好奇心驱动的无监督数据收集方法，通过自适应的时间距离扩展特征空间，从而提高多任务离线强化学习的效率和能力。该方法通过估计 k 步未来状态从当前状态可达的概率，并调整动力学模型预测的未来步数，实现自适应可达性机制，从而实现特征表示的多样化，使智能体能够通过好奇心导航自己收集更高质量的数据。经实验证明，该方法在 DeepMind 控制套件的各种下游离线强化学习任务中超越了现有的无监督方法，在效率和学习性能上表现出色。