Current advances in recommender systems have been remarkably successful in
optimizing immediate engagement. However, long-term user engagement, a more
desirable performance metric, remains difficult to improve. Meanwhile, recent
reinforcement learning (RL) algorithms have shown their effectiveness in a
variety of long-term goal optimization tasks. For this reason, RL is widely
considered as a promising framework for optimizing long-term user engagement in
recommendation. Despite being a promising approach, the application of RL
heavily relies on well-designed rewards, but designing rewards related to
long-term user engagement is quite difficult. To mitigate the problem, we
propose a novel paradigm, Preference-based Recommender systems (PrefRec), which
allows RL recommender systems to learn from preferences about users' historical
behaviors rather than explicitly defined rewards. Such preferences are easily
accessible through techniques such as crowdsourcing, as they do not require any
expert knowledge. With PrefRec, we can fully exploit the advantages of RL in
optimizing long-term goals, while avoiding complex reward engineering. PrefRec
uses the preferences to automatically train a reward function in an end-to-end
manner. The reward function is then used to generate learning signals to train
the recommendation policy. Furthermore, we design an effective optimization
method for PrefRec, which uses an additional value function, expectile
regression and reward model pre-training to improve the performance. Extensive
experiments are conducted on a variety of long-term user engagement
optimization tasks. The results show that PrefRec significantly outperforms
previous state-of-the-art methods in all the tasks.

本文提出了一种名为 PrefRec 的新模型，它基于用户历史行为的喜好训练基于强化学习的推荐器，可有效地优化长期用户参与度，在大量的实验中，PrefRec 在所有任务中显着优于之前的最先进方法。