We introduce a generalization of temporal-difference (TD) learning to
networks of interrelated predictions. Rather than relating a single prediction
to itself at a later time, as in conventional TD methods, a TD network relates
each prediction in a set of predictions to other predictions in the set at a
later time. TD networks can represent and apply TD learning to a much wider
class of predictions than has previously been possible. Using a random-walk
example, we show that these networks can be used to learn to predict by a fixed
interval, which is not possible with conventional TD methods. Secondly, we show
that if the inter-predictive relationships are made conditional on action, then
the usual learning-efficiency advantage of TD methods over Monte Carlo
(supervised learning) methods becomes particularly pronounced. Thirdly, we
demonstrate that TD networks can learn predictive state representations that
enable exact solution of a non-Markov problem. A very broad range of
inter-predictive temporal relationships can be expressed in these networks.
Overall we argue that TD networks represent a substantial extension of the
abilities of TD methods and bring us closer to the goal of representing world
knowledge in entirely predictive, grounded terms.

介绍了一种将时间差异（TD）学习推广到相互关联预测网络的方法， TD 网络能够表示和应用 TD 学习到比以前更广泛的预测类别，并通过将预测之间的关系作为条件来提高学习效率，此外，还演示了 TD 网络可以学习预测状态表示，成为 TD 方法能力的实质性扩展之一，带我们更加接近以完全预测和基于经验的方式表达世界知识的目标。