We introduce learning and planning algorithms for average-reward MDPs,
including 1) the first general proven-convergent off-policy model-free control
algorithm without reference states, 2) the first proven-convergent off-policy
model-free prediction algorithm, and 3) the first off-policy learning algorithm
that converges to the actual value function rather than to the value function
plus an offset. All of our algorithms are based on using the
temporal-difference error rather than the conventional error when updating the
estimate of the average reward. Our proof techniques are a slight
generalization of those by Abounadi, Bertsekas, and Borkar (2001). In
experiments with an Access-Control Queuing Task, we show some of the
difficulties that can arise when using methods that rely on reference states
and argue that our new algorithms can be significantly easier to use.

本研究提出了一种基于平均报酬 MDPs 的学习和规划算法，其中包括第一种无参考状态的普遍证明收敛的无模型控制算法、第一个证明收敛的无政策自由预测算法，以及第一个离线学习算法，其收敛于实际值函数而不是值函数增加一个偏移量。在使用时间差错错误而不是常规错误更新平均报酬估计时，我们的所有算法都基于此。