Industrial recommender systems deal with extremely large action spaces -- many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (e.g., user clicks, dwell time) are available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at Youtube, built with a policy-gradient-based algorithm, i.e. REINFORCE. The contributions of the paper are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions; (2) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our approaches through a series of simulations and multiple live experiments on Youtube.

本文提出了一种方法，在工业推荐系统中使用基于策略梯度的REINFORCE算法来解决通过学习从之前版本的推荐中观察到的数据偏差，同时通过纠正离线反馈数据的偏差来消除REINFORCE在大规模action space中的数据偏差，同时提出一种针对推荐多个项目的新型top-K保真度校正方法，并通过模拟和多个实验表明了该方法的有效性。

一种用于 REINFORCE 推荐系统的 Top-K 离线校正算法