Offline reinforcement learning (RL) methods can generally be categorized into
two types: RL-based and Imitation-based. RL-based methods could in principle
enjoy out-of-distribution generalization but suffer from erroneous off-policy
evaluation. Imitation-based methods avoid off-policy evaluation but are too
conservative to surpass the dataset. In this study, we propose an alternative
approach, inheriting the training stability of imitation-style methods while
still allowing logical out-of-distribution generalization. We decompose the
conventional reward-maximizing policy in offline RL into a guide-policy and an
execute-policy. During training, the guide-poicy and execute-policy are learned
using only data from the dataset, in a supervised and decoupled manner. During
evaluation, the guide-policy guides the execute-policy by telling where it
should go so that the reward can be maximized, serving as the \textit{Prophet}.
By doing so, our algorithm allows \textit{state-compositionality} from the
dataset, rather than \textit{action-compositionality} conducted in prior
imitation-style methods. We dumb this new approach Policy-guided Offline RL
(\texttt{POR}). \texttt{POR} demonstrates the state-of-the-art performance on
D4RL, a standard benchmark for offline RL. We also highlight the benefits of
\texttt{POR} in terms of improving with supplementary suboptimal data and
easily adapting to new tasks by only changing the guide-poicy.

该研究提出了一种 Policy-guided Offline RL 算法，该算法在训练时将想法分解为指导策略和执行策略，并通过指导策略来指导执行策略以实现状态组合性。该算法在离线 RL 的标准基准 D4RL 上展示了最高效的性能，并可以通过改变指导策略来轻松适应新的任务。