Offline reinforcement learning (RL) is a compelling paradigm to extend RL's
practical utility by leveraging pre-collected, static datasets, thereby
avoiding the limitations associated with collecting online interactions. The
major difficulty in offline RL is mitigating the impact of approximation errors
when encountering out-of-distribution (OOD) actions; doing so ineffectively
will lead to policies that prefer OOD actions, which can lead to unexpected and
potentially catastrophic results. Despite the variety of works proposed to
address this issue, they tend to excessively suppress the value function in and
around OOD regions, resulting in overly pessimistic value estimates. In this
paper, we propose a novel framework called Strategically Conservative
Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to
estimate, ultimately resulting in less conservative value estimates. Our
approach exploits the inherent strengths of neural networks to interpolate,
while carefully navigating their limitations in extrapolation, to obtain
pessimistic yet still property calibrated value estimates. Theoretical analysis
also shows that the value function learned by SCQ is still conservative, but
potentially much less so than that of Conservative Q-learning (CQL). Finally,
extensive evaluation on the D4RL benchmark tasks shows our proposed method
outperforms state-of-the-art methods. Our code is available through
https://github.com/purewater0901/SCQ.

提出了一种名为 Strategically Conservative Q-Learning (SCQ) 的新框架，用于解决离线强化学习中的逼近误差和分布外动作的问题，在 D4RL 基准任务上表现优于现有方法。

策略性保守型 Q 学习

Strategically Conservative Q-Learning

Offline reinforcement learning (RL), which aims to learn an optimal policy
using a previously collected static dataset, is an important paradigm of RL.
Standard RL methods often perform poorly in this regime due to the function
approximation errors on out-of-distribution actions. While a variety of
regularization methods have been proposed to mitigate this issue, they are
often constrained by policy classes with limited expressiveness that can lead
to highly suboptimal solutions. In this paper, we propose representing the
policy as a diffusion model, a recent class of highly-expressive deep
generative models. We introduce Diffusion Q-learning (Diffusion-QL) that
utilizes a conditional diffusion model to represent the policy. In our
approach, we learn an action-value function and we add a term maximizing
action-values into the training loss of the conditional diffusion model, which
results in a loss that seeks optimal actions that are near the behavior policy.
We show the expressiveness of the diffusion model-based policy, and the
coupling of the behavior cloning and policy improvement under the diffusion
model both contribute to the outstanding performance of Diffusion-QL. We
illustrate the superiority of our method compared to prior works in a simple 2D
bandit example with a multimodal behavior policy. We then show that our method
can achieve state-of-the-art performance on the majority of the D4RL benchmark
tasks.

本文提出了一种利用扩散模型表示策略的离线强化学习方法 (Diffusion Q-learning)，与行为克隆和策略改进的耦合均有助于实现出色的性能，证明了该方法在大多数 D4RL 基准任务中表现出卓越的性能。