Effective offline RL methods require properly handling out-of-distribution
actions. Implicit Q-learning (IQL) addresses this by training a Q-function
using only dataset actions through a modified Bellman backup. However, it is
unclear which policy actually attains the values represented by this implicitly
trained Q-function. In this paper, we reinterpret IQL as an actor-critic method
by generalizing the critic objective and connecting it to a
behavior-regularized implicit actor. This generalization shows how the induced
actor balances reward maximization and divergence from the behavior policy,
with the specific loss choice determining the nature of this tradeoff. Notably,
this actor can exhibit complex and multimodal characteristics, suggesting
issues with the conditional Gaussian actor fit with advantage weighted
regression (AWR) used in prior methods. Instead, we propose using samples from
a diffusion parameterized behavior policy and weights computed from the critic
to then importance sampled our intended policy. We introduce Implicit Diffusion
Q-learning (IDQL), combining our general IQL critic with the policy extraction
method. IDQL maintains the ease of implementation of IQL while outperforming
prior offline RL methods and demonstrating robustness to hyperparameters. Code
is available at this https URL

使用 Diffusion parameterized behavior policy 和 Implicit Q-learning (IQL) 模型，提出了一个新的 actor-critic 模型，称为 Implicit Diffusion Q-learning (IDQL)，能够处理 offline RL 问题中的 out-of-distribution actions。