Batch reinforcement learning (RL) is important to apply RL algorithms to many
high stakes tasks. Doing batch RL in a way that yields a reliable new policy in
large domains is challenging: a new decision policy may visit states and
actions outside the support of the batch data, and function approximation and
optimization with limited samples can further increase the potential of
learning policies with overly optimistic estimates of their future performance.
Recent algorithms have shown promise but can still be overly optimistic in
their expected outcomes. Theoretical work that provides strong guarantees on
the performance of the output policy relies on a strong concentrability
assumption, that makes it unsuitable for cases where the ratio between
state-action distributions of behavior policy and some candidate policies is
large. This is because in the traditional analysis, the error bound scales up
with this ratio. We show that a small modification to Bellman optimality and
evaluation back-up to take a more conservative update can have much stronger
guarantees. In certain settings, they can find the approximately best policy
within the state-action space explored by the batch data, without requiring a
priori assumptions of concentrability. We highlight the necessity of our
conservative update and the limitations of previous algorithms and analyses by
illustrative MDP examples, and demonstrate an empirical comparison of our
algorithm and other state-of-the-art batch RL baselines in standard benchmarks.

本文介绍了一种基于 Bellman 备份的批量强化学习算法，它采用一种更加保守的更新策略来提高输出策略的性能保证，并通过演示 MDP 示例和在标准基准测试中的实证比较来突出了我们保守更新的必要性和以前算法和分析的局限性。