Temporal credit assignment in reinforcement learning is challenging due to
delayed and stochastic outcomes. Monte Carlo targets can bridge long delays
between action and consequence but lead to high-variance targets due to
stochasticity. Temporal difference (TD) learning uses bootstrapping to overcome
variance but introduces a bias that can only be corrected through many
iterations. TD($\lambda$) provides a mechanism to navigate this bias-variance
tradeoff smoothly. Appropriately selecting $\lambda$ can significantly improve
performance. Here, we propose Chunked-TD, which uses predicted probabilities of
transitions from a model for computing $\lambda$-return targets. Unlike other
model-based solutions to credit assignment, Chunked-TD is less vulnerable to
model inaccuracies. Our approach is motivated by the principle of history
compression and 'chunks' trajectories for conventional TD learning. Chunking
with learned world models compresses near-deterministic regions of the
environment-policy interaction to speed up credit assignment while still
bootstrapping when necessary. We propose algorithms that can be implemented
online and show that they solve some problems much faster than conventional
TD($\lambda$).

时间差分学习中的时间分配问题是具有挑战性的，本文提出了一种名为 Chunked-TD 的算法，它使用来自模型的预测概率来计算 lambda 回报目标，相对于其他基于模型的解决方案，Chunked-TD 更不容易受到模型的不准确性影响，该算法可以在线实施，解决了某些问题比传统的 TD (lambda) 算法更快。