Predicting and reasoning about the future lie at the heart of many
time-series questions. For example, goal-conditioned reinforcement learning can
be viewed as learning representations to predict which states are likely to be
visited in the future. While prior methods have used contrastive predictive
coding to model time series data, learning representations that encode
long-term dependencies usually requires large amounts of data. In this paper,
we introduce a temporal difference version of contrastive predictive coding
that stitches together pieces of different time series data to decrease the
amount of data required to learn predictions of future events. We apply this
representation learning method to derive an off-policy algorithm for
goal-conditioned RL. Experiments demonstrate that, compared with prior RL
methods, ours achieves $2 \times$ median improvement in success rates and can
better cope with stochastic environments. In tabular settings, we show that our
method is about $20 \times$ more sample efficient than the successor
representation and $1500 \times$ more sample efficient than the standard (Monte
Carlo) version of contrastive predictive coding.

本文介绍了一种自适应时间差异版对比性预测编码，通过拼接不同时间序列数据的片段来减少学习未来事件预测所需的数据量，并将该方法应用于推导目标条件强化学习的离线策略算法。实验表明，与先前的强化学习方法相比，我们的方法在成功率方面取得了 2 倍的中值提高，能够更好地应对随机环境。在表格设置中，我们展示了我们的方法比后续表示方法更节约样本，比标准（蒙特卡洛）版本的对比性预测编码更节约样本数量约 20 倍和 1500 倍。