Effectively leveraging large, previously collected datasets in reinforcement
learning (RL) is a key challenge for large-scale real-world applications.
Offline RL algorithms promise to learn effective policies from
previously-collected, static datasets without further interaction. However, in
practice, offline RL presents a major challenge, and standard off-policy RL
methods can fail due to overestimation of values induced by the distributional
shift between the dataset and the learned policy, especially when training on
complex and multi-modal data distributions. In this paper, we propose
conservative Q-learning (CQL), which aims to address these limitations by
learning a conservative Q-function such that the expected value of a policy
under this Q-function lower-bounds its true value. We theoretically show that
CQL produces a lower bound on the value of the current policy and that it can
be incorporated into a policy learning procedure with theoretical improvement
guarantees. In practice, CQL augments the standard Bellman error objective with
a simple Q-value regularizer which is straightforward to implement on top of
existing deep Q-learning and actor-critic implementations. On both discrete and
continuous control domains, we show that CQL substantially outperforms existing
offline RL methods, often learning policies that attain 2-5 times higher final
return, especially when learning from complex and multi-modal data
distributions.

本论文提出了保守型 Q-learning（CQL），通过学习保守型 Q 函数以得到预期值，有效地解决了离线强化学习（offline RL）中的价值估计问题，从而提高了学习性能。在实验中，我们将 CQL 应用于复杂和多模态数据分布，证明其在离线 RL 方法中的优越性，能学习到比现有离线 RL 方法 2 到 5 倍更高的最终回报的策略