Pre-training with offline data and online fine-tuning using reinforcement
learning is a promising strategy for learning control policies by leveraging
the best of both worlds in terms of sample efficiency and performance. One
natural approach is to initialize the policy for online learning with the one
trained offline. In this work, we introduce a policy expansion scheme for this
task. After learning the offline policy, we use it as one candidate policy in a
policy set. We then expand the policy set with another policy which will be
responsible for further learning. The two policies will be composed in an
adaptive manner for interacting with the environment. With this approach, the
policy previously learned offline is fully retained during online learning,
thus mitigating the potential issues such as destroying the useful behaviors of
the offline policy in the initial stage of online learning while allowing the
offline policy participate in the exploration naturally in an adaptive manner.
Moreover, new useful behaviors can potentially be captured by the newly added
policy through learning. Experiments are conducted on a number of tasks and the
results demonstrate the effectiveness of the proposed approach.

通过先预训练离线数据，再使用强化学习进行在线微调是一种有效的控制策略学习策略，本文提出了一种策略扩展方案以增加新的策略来参与探索，有效提高了学习效率和性能表现。