Offline reinforcement learning (RL) aims to learn a policy using only
pre-collected and fixed data. Although avoiding the time-consuming online
interactions in RL, it poses challenges for out-of-distribution (OOD) state
actions and often suffers from data inefficiency for training. Despite many
efforts being devoted to addressing OOD state actions, the latter (data
inefficiency) receives little attention in offline RL. To address this, this
paper proposes the cross-domain offline RL, which assumes offline data
incorporate additional source-domain data from varying transition dynamics
(environments), and expects it to contribute to the offline data efficiency. To
do so, we identify a new challenge of OOD transition dynamics, beyond the
common OOD state actions issue, when utilizing cross-domain offline data. Then,
we propose our method BOSA, which employs two support-constrained objectives to
address the above OOD issues. Through extensive experiments in the cross-domain
offline RL setting, we demonstrate BOSA can greatly improve offline data
efficiency: using only 10\% of the target data, BOSA could achieve {74.4\%} of
the SOTA offline RL performance that uses 100\% of the target data.
Additionally, we also show BOSA can be effortlessly plugged into model-based
offline RL and noising data augmentation techniques (used for generating
source-domain data), which naturally avoids the potential dynamics mismatch
between target-domain data and newly generated source-domain data.

本文提出了交叉领域离线强化学习方法 BOSA，通过利用来自不同转移动态（环境）的源领域数据，解决了离线 RL 中存在的数据效率不高的问题，并通过实验证明其在离线数据效率方面的有效性。