Offline reinforcement learning (RL) defines the task of learning from a fixed
batch of data. Due to errors in value estimation from out-of-distribution
actions, most offline RL algorithms take the approach of constraining or
regularizing the policy with the actions contained in the dataset. Built on
pre-existing RL algorithms, modifications to make an RL algorithm work offline
comes at the cost of additional complexity. Offline RL algorithms introduce new
hyperparameters and often leverage secondary components such as generative
models, while adjusting the underlying RL algorithm. In this paper we aim to
make a deep RL algorithm work while making minimal changes. We find that we can
match the performance of state-of-the-art offline RL algorithms by simply
adding a behavior cloning term to the policy update of an online RL algorithm
and normalizing the data. The resulting algorithm is a simple to implement and
tune baseline, while more than halving the overall run time by removing the
additional computational overhead of previous methods.

通过在在线强化学习算法的策略更新中添加行为克隆项并规范化数据，在保持简单性的同时，最大限度地提高了运行效率，从而实现了与现有离线 RL 算法相当的性能。