In this paper, we introduce a fresh perspective on the challenges of credit
assignment and policy evaluation. First, we delve into the nuances of
eligibility traces and explore instances where their updates may result in
unexpected credit assignment to preceding states. From this investigation
emerges the concept of a novel value function, which we refer to as the
\emph{bidirectional value function}. Unlike traditional state value functions,
bidirectional value functions account for both future expected returns (rewards
anticipated from the current state onward) and past expected returns
(cumulative rewards from the episode's start to the present). We derive
principled update equations to learn this value function and, through
experimentation, demonstrate its efficacy in enhancing the process of policy
evaluation. In particular, our results indicate that the proposed learning
approach can, in certain challenging contexts, perform policy evaluation more
rapidly than TD($\lambda$) -- a method that learns forward value functions,
$v^\pi$, \emph{directly}. Overall, our findings present a new perspective on
eligibility traces and potential advantages associated with the novel value
function it inspires, especially for policy evaluation.

我们介绍了对信用分配和政策评估挑战的新视角，并引入了双向值函数的概念，它可以同时考虑未来期望回报和过去累计回报，通过实验证明这种价值函数在增强政策评估过程中的有效性。