Recommending novel content, which expands user horizons by introducing them
to new interests, has been shown to improve users' long-term experience on
recommendation platforms \cite{chen2021values}. Users however are not
constantly looking to explore novel content. It is therefore crucial to
understand their novelty-seeking intent and adjust the recommendation policy
accordingly. Most existing literature models a user's propensity to choose
novel content or to prefer a more diverse set of recommendations at individual
interactions. Hierarchical structure, on the other hand, exists in a user's
novelty-seeking intent, which is manifested as a static and intrinsic user
preference for seeking novelty along with a dynamic session-based propensity.
To this end, we propose a novel hierarchical reinforcement learning-based
method to model the hierarchical user novelty-seeking intent, and to adapt the
recommendation policy accordingly based on the extracted user novelty-seeking
propensity. We further incorporate diversity and novelty-related measurement in
the reward function of the hierarchical RL (HRL) agent to encourage user
exploration \cite{chen2021values}. We demonstrate the benefits of explicitly
modeling hierarchical user novelty-seeking intent in recommendations through
extensive experiments on simulated and real-world datasets. In particular, we
demonstrate that the effectiveness of our proposed hierarchical RL-based method
lies in its ability to capture such hierarchically-structured intent. As a
result, the proposed HRL model achieves superior performance on several public
datasets, compared with state-of-art baselines.

提出了一种新的基于层次强化学习的方法，用于建模用户的层次式好奇心意图，从而可以根据提取的用户好奇心倾向来调整推荐策略，通过在奖励功能中引入多样性和新颖性相关的度量来鼓励用户的探索，通过针对模拟和实际数据集的大量实验来证明了该方法的效果优于现有的基线模型。

Hierarchical Reinforcement Learning for Modeling User Novelty-Seeking  Intent in Recommender Systems

Large language models (LLMs) have shown impressive capabilities in natural
language understanding and generation. Their potential for deeper user
understanding and improved personalized user experience on recommendation
platforms is, however, largely untapped. This paper aims to address this gap.
Recommender systems today capture users' interests through encoding their
historical activities on the platforms. The generated user representations are
hard to examine or interpret. On the other hand, if we were to ask people about
interests they pursue in their life, they might talk about their hobbies, like
I just started learning the ukulele, or their relaxation routines, e.g., I like
to watch Saturday Night Live, or I want to plant a vertical garden. We argue,
and demonstrate through extensive experiments, that LLMs as foundation models
can reason through user activities, and describe their interests in nuanced and
interesting ways, similar to how a human would.
We define interest journeys as the persistent and overarching user interests,
in other words, the non-transient ones. These are the interests that we believe
will benefit most from the nuanced and personalized descriptions. We introduce
a framework in which we first perform personalized extraction of interest
journeys, and then summarize the extracted journeys via LLMs, using techniques
like few-shot prompting, prompt-tuning and fine-tuning. Together, our results
in prompting LLMs to name extracted user journeys in a large-scale industrial
platform demonstrate great potential of these models in providing deeper, more
interpretable, and controllable user understanding. We believe LLM powered user
understanding can be a stepping stone to entirely new user experiences on
recommendation platforms that are journey-aware, assistive, and enabling
frictionless conversation down the line.

这篇文章旨在通过大型语言模型对用户活动进行细致和透彻的理解，从而提高推荐平台上用户的个性化体验和更深层次的理解。通过提取兴趣轨迹并通过大型语言模型进行描述，我们证明了这种方法可以为用户提供深入、可解释和可控的理解，为用户体验提供新的可能性。

用户兴趣旅程的大型语言模型

Large Language Models for User Interest Journeys

With the recent prevalence of Reinforcement Learning (RL), there have been
tremendous interests in utilizing RL for online advertising in recommendation
platforms (e.g., e-commerce and news feed sites). However, most RL-based
advertising algorithms focus on optimizing ads' revenue while ignoring the
possible negative influence of ads on user experience of recommended items
(products, articles and videos). Developing an optimal advertising algorithm in
recommendations faces immense challenges because interpolating ads improperly
or too frequently may decrease user experience, while interpolating fewer ads
will reduce the advertising revenue. Thus, in this paper, we propose a novel
advertising strategy for the rec/ads trade-off. To be specific, we develop an
RL-based framework that can continuously update its advertising strategies and
maximize reward in the long run. Given a recommendation list, we design a novel
Deep Q-network architecture that can determine three internally related tasks
jointly, i.e., (i) whether to interpolate an ad or not in the recommendation
list, and if yes, (ii) the optimal ad and (iii) the optimal location to
interpolate. The experimental results based on real-world data demonstrate the
effectiveness of the proposed framework.

本文提出了一种基于强化学习的广告策略，通过设计一个新型的深度 Q 网络结构，可以连续更新其广告策略，在推荐列表中插入广告或不插入广告，并选择最优广告和最佳插入位置，使得推荐广告效果显著提升。