We investigate projection methods, for evaluating a linear approximation of the value function of a policy in a Markov Decision Process context. We consider two popular approaches, the one-step Temporal Difference fix-point computation (TD(0)) and the Bellman Residual (BR) minimization. We describe examples, where each method outperforms the other. We highlight a simple relation between the objective function they minimize, and show that while BR enjoys a performance guarantee, TD(0) does not in general. We then propose a unified view in terms of oblique projections of the Bellman equation, which substantially simplifies and extends the characterization of (schoknecht,2002) and the recent analysis of (Yu & Bertsekas, 2008). Eventually, we describe some simulations that suggest that if the TD(0) solution is usually slightly better than the BR solution, its inherent numerical instability makes it very bad in some cases, and thus worse on average.

研究了马尔可夫决策过程情景下政策价值函数的线性逼近的投影方法。分别考虑了一步时序差分计算（TD（0））和Bellman残差（BR）最小化的两种流行方法，并描述了它们的性能优劣。最终，提出了一个统一的观点，揭示了零部件投影在Bellman等式上的作用。

计算时间差异的定点还是最小化贝尔曼残差？统一的斜投影视角