Pretraining RL models on offline video datasets is a promising way to improve
their training efficiency in online tasks, but challenging due to the inherent
mismatch in tasks, dynamics, and behaviors across domains. A recent model, APV,
sidesteps the accompanied action records in offline datasets and instead
focuses on pretraining a task-irrelevant, action-free world model within the
source domains. We present Vid2Act, a model-based RL method that learns to
transfer valuable action-conditioned dynamics and potentially useful action
demonstrations from offline to online settings. The main idea is to use the
world models not only as simulators for behavior learning but also as tools to
measure the domain relevance for both dynamics representation transfer and
policy transfer. Specifically, we train the world models to generate a set of
time-varying task similarities using a domain-selective knowledge distillation
loss. These similarities serve two purposes: (i) adaptively transferring the
most useful source knowledge to facilitate dynamics learning, and (ii) learning
to replay the most relevant source actions to guide the target policy. We
demonstrate the advantages of Vid2Act over the action-free visual RL
pretraining method in both Meta-World and DeepMind Control Suite.

Vid2Act 是一种基于模型的强化学习方法，其使用世界模型作为行为学习的模拟器并使用它们来衡量动力学表示转移和策略转移的域相关性，以将有价值的动作条件动态和潜在有用的行动演示从离线到在线环境进行转移。