Open-world novelty--a sudden change in the mechanics or properties of an
environment--is a common occurrence in the real world. Novelty adaptation is an
agent's ability to improve its policy performance post-novelty. Most
reinforcement learning (RL) methods assume that the world is a closed, fixed
process. Consequentially, RL policies adapt inefficiently to novelties. To
address this, we introduce WorldCloner, an end-to-end trainable neuro-symbolic
world model for rapid novelty adaptation. WorldCloner learns an efficient
symbolic representation of the pre-novelty environment transitions, and uses
this transition model to detect novelty and efficiently adapt to novelty in a
single-shot fashion. Additionally, WorldCloner augments the policy learning
process using imagination-based adaptation, where the world model simulates
transitions of the post-novelty environment to help the policy adapt. By
blending ''imagined'' transitions with interactions in the post-novelty
environment, performance can be recovered with fewer total environment
interactions. Using environments designed for studying novelty in sequential
decision-making problems, we show that the symbolic world model helps its
neural policy adapt more efficiently than model-based and model-based
neural-only reinforcement learning methods.

本文研究了如何提高强化学习算法的适应性，通过引入一种名为 WorldCloner 的训练神经符号世界模型，从而实现快速的新颖性适应，并使用想象力来辅助适应后的策略，达到更高的效率。