Reinforcement learning (RL) has been widely applied in recommendation systems
due to its potential in optimizing the long-term engagement of users. From the
perspective of RL, recommendation can be formulated as a Markov decision
process (MDP), where recommendation system (agent) can interact with users
(environment) and acquire feedback (reward signals).However, it is impractical
to conduct online interactions with the concern on user experience and
implementation complexity, and we can only train RL recommenders with offline
datasets containing limited reward signals and state transitions. Therefore,
the data sparsity issue of reward signals and state transitions is very severe,
while it has long been overlooked by existing RL recommenders.Worse still, RL
methods learn through the trial-and-error mode, but negative feedback cannot be
obtained in implicit feedback recommendation tasks, which aggravates the
overestimation problem of offline RL recommender. To address these challenges,
we propose a novel RL recommender named model-enhanced contrastive
reinforcement learning (MCRL). On the one hand, we learn a value function to
estimate the long-term engagement of users, together with a conservative value
learning mechanism to alleviate the overestimation problem.On the other hand,
we construct some positive and negative state-action pairs to model the reward
function and state transition function with contrastive learning to exploit the
internal structure information of MDP. Experiments demonstrate that the
proposed method significantly outperforms existing offline RL and
self-supervised RL methods with different representative backbone networks on
two real-world datasets.

通过提出一种名为模型增强对比强化学习（MCRL）的新型强化学习推荐器，我们解决了离线数据集的稀疏性以及负反馈无法获取的问题，并通过对比学习来优化奖励函数和状态转移函数，实验结果表明该方法在两个现实世界数据集上显著优于现有的离线强化学习和自监督强化学习方法。