Offline reinforcement learning (RL) is a compelling framework for learning
optimal policies from past experiences without additional interaction with the
environment. Nevertheless, offline RL inevitably faces the problem of
distributional shifts, where the states and actions encountered during policy
execution may not be in the training dataset distribution. A common solution
involves incorporating conservatism into the policy or the value function to
safeguard against uncertainties and unknowns. In this work, we focus on
achieving the same objectives of conservatism but from a different perspective.
We propose COmpositional COnservatism with Anchor-seeking (COCOA) for offline
RL, an approach that pursues conservatism in a compositional manner on top of
the transductive reparameterization (Netanyahu et al., 2023), which decomposes
the input variable (the state in our case) into an anchor and its difference
from the original input. Our COCOA seeks both in-distribution anchors and
differences by utilizing the learned reverse dynamics model, encouraging
conservatism in the compositional input space for the policy or value function.
Such compositional conservatism is independent of and agnostic to the prevalent
behavioral conservatism in offline RL. We apply COCOA to four state-of-the-art
offline RL algorithms and evaluate them on the D4RL benchmark, where COCOA
generally improves the performance of each algorithm. The code is available at
this https URL

离线强化学习中的分布变化问题通过 COmpositional COnservatism with Anchor-seeking (COCOA) 方法得以解决，该方法在复杂的输入空间中寻找共享分布点和差异，从而提高算法性能。