Recommender systems are a ubiquitous feature of online platforms.
Increasingly, they are explicitly tasked with increasing users' long-term
satisfaction. In this context, we study a content exploration task, which we
formalize as a multi-armed bandit problem with delayed rewards. We observe that
there is an apparent trade-off in choosing the learning signal: Waiting for the
full reward to become available might take several weeks, hurting the rate at
which learning happens, whereas measuring short-term proxy rewards reflects the
actual long-term goal only imperfectly. We address this challenge in two steps.
First, we develop a predictive model of delayed rewards that incorporates all
information obtained to date. Full observations as well as partial (short or
medium-term) outcomes are combined through a Bayesian filter to obtain a
probabilistic belief. Second, we devise a bandit algorithm that takes advantage
of this new predictive model. The algorithm quickly learns to identify content
aligned with long-term success by carefully balancing exploration and
exploitation. We apply our approach to a podcast recommendation problem, where
we seek to identify shows that users engage with repeatedly over two months. We
empirically validate that our approach results in substantially better
performance compared to approaches that either optimize for short-term proxies,
or wait for the long-term outcome to be fully realized.

在在线平台中，推荐系统是一个普遍存在的功能，越来越多地被明确要求增加用户的长期满意度。本研究主要关注内容探索任务，将其形式化为一个带有延迟奖励的多臂赌博问题。我们观察到在选择学习信号时存在明显的权衡：等待全部奖励可用可能需要几个星期，从而影响学习的速度，而测量短期代理奖励只会不完美地反映实际的长期目标。我们通过两个步骤来解决这个挑战。首先，我们开发了一个预测延迟奖励的模型，该模型综合了迄今为止获得的所有信息。通过贝叶斯滤波器将完整观测和部分观测（短期或中期结果）结合起来，得到概率信念。其次，我们设计了一种利用这个新预测模型的赌博算法。该算法通过巧妙地平衡探索和开发快速学习到与长期成功一致的内容。我们将我们的方法应用于播客推荐问题，期望在两个月内找到用户重复参与的节目。经验证明，我们的方法相比于优化短期代理或等待完全实现长期结果的方法，能够显著提高性能。