Traditional recommendation setting tends to excessively cater to users'
immediate interests and neglect their long-term engagement. To address it, it
is crucial to incorporate planning capabilities into the recommendation
decision-making process to develop policies that take into account both
immediate interests and long-term engagement. Despite Reinforcement Learning
(RL) can learn planning capacity by maximizing cumulative reward, the scarcity
of recommendation data presents challenges such as instability and
susceptibility to overfitting when training RL models from scratch.
In this context, we propose to leverage the remarkable planning capabilities
over sparse data of Large Language Models (LLMs) for long-term recommendation.
The key lies in enabling a language model to understand and apply task-solving
principles effectively in personalized recommendation scenarios, as the model's
pre-training may not naturally encompass these principles, necessitating the
need to inspire or teach the model. To achieve this, we propose a Bi-level
Learnable LLM Planner framework, which combines macro-learning and
micro-learning through a hierarchical mechanism. The framework includes a
Planner and Reflector for acquiring high-level guiding principles and an
Actor-Critic component for planning personalization. Extensive experiments
validate the superiority of the framework in learning to plan for long-term
recommendations.

利用大型语言模型的规划能力，提出一种可学习的长期推荐规划框架，结合强化学习和层次学习。实验证实该框架在学习长期推荐规划方面具有优势。

利用双层可学习大语言模型规划增强长期推荐

Enhancing Long-Term Recommendation with Bi-level Learnable Large  Language Model Planning

Most practical recommender systems focus on estimating immediate user
engagement without considering the long-term effects of recommendations on user
behavior. Reinforcement learning (RL) methods offer the potential to optimize
recommendations for long-term user engagement. However, since users are often
presented with slates of multiple items - which may have interacting effects on
user choice - methods are required to deal with the combinatorics of the RL
action space. In this work, we address the challenge of making slate-based
recommendations to optimize long-term value using RL. Our contributions are
three-fold. (i) We develop SLATEQ, a decomposition of value-based
temporal-difference and Q-learning that renders RL tractable with slates. Under
mild assumptions on user choice behavior, we show that the long-term value
(LTV) of a slate can be decomposed into a tractable function of its component
item-wise LTVs. (ii) We outline a methodology that leverages existing myopic
learning-based recommenders to quickly develop a recommender that handles LTV.
(iii) We demonstrate our methods in simulation, and validate the scalability of
decomposed TD-learning using SLATEQ in live experiments on YouTube.

该研究提供了使用深度强化学习技术解决个性化推荐系统中长期用户参与度问题的方法，通过分解价值函数，考虑了物品组合效应，并实验证明了该方法的可行性和扩展性。