Widely-used deep reinforcement learning algorithms have been shown to fail in
the batch setting--learning from a fixed data set without interaction with the
environment. Following this result, there have been several papers showing
reasonable performances under a variety of environments and batch settings. In
this paper, we benchmark the performance of recent off-policy and batch
reinforcement learning algorithms under unified settings on the Atari domain,
with data generated by a single partially-trained behavioral policy. We find
that under these conditions, many of these algorithms underperform DQN trained
online with the same amount of data, as well as the partially-trained
behavioral policy. To introduce a strong baseline, we adapt the
Batch-Constrained Q-learning algorithm to a discrete-action setting, and show
it outperforms all existing algorithms at this task.

本研究在 Atari 领域中利用单个部分训练的行为策略生成的数据对最近的离线策略和批量强化学习算法的性能进行了基准测试，并发现在这些条件下，许多算法表现不佳，而 Batch-Constrained Q-learning 算法适应离散动作环境后在这项任务中表现最佳。