User modeling, which learns to represent users into a low-dimensional
representation space based on their past behaviors, got a surge of interest
from the industry for providing personalized services to users. Previous
efforts in user modeling mainly focus on learning a task-specific user
representation that is designed for a single task. However, since learning
task-specific user representations for every task is infeasible, recent studies
introduce the concept of universal user representation, which is a more
generalized representation of a user that is relevant to a variety of tasks.
Despite their effectiveness, existing approaches for learning universal user
representations are impractical in real-world applications due to the data
requirement, catastrophic forgetting and the limited learning capability for
continually added tasks. In this paper, we propose a novel continual user
representation learning method, called TERACON, whose learning capability is
not limited as the number of learned tasks increases while capturing the
relationship between the tasks. The main idea is to introduce an embedding for
each task, i.e., task embedding, which is utilized to generate task-specific
soft masks that not only allow the entire model parameters to be updated until
the end of training sequence, but also facilitate the relationship between the
tasks to be captured. Moreover, we introduce a novel knowledge retention module
with pseudo-labeling strategy that successfully alleviates the long-standing
problem of continual learning, i.e., catastrophic forgetting. Extensive
experiments on public and proprietary real-world datasets demonstrate the
superiority and practicality of TERACON. Our code is available at
this https URL

本文提出了一种名为 TERACON 的新型连续用户表示学习方法，它的学习能力随着学习任务的增加而不受限制，同时捕捉任务之间的关系，通过任务嵌入生成任务特定的软掩码，利用伪标签策略和知识保留模块成功缓解了连续学习的问题。

任务关系感知的连续用户表示学习

Task Relation-aware Continual User Representation Learning

User modeling is critical for many personalized web services. Many existing
methods model users based on their behaviors and the labeled data of target
tasks. However, these methods cannot exploit useful information in unlabeled
user behavior data, and their performance may be not optimal when labeled data
is scarce. Motivated by pre-trained language models which are pre-trained on
large-scale unlabeled corpus to empower many downstream tasks, in this paper we
propose to pre-train user models from large-scale unlabeled user behaviors
data. We propose two self-supervision tasks for user model pre-training. The
first one is masked behavior prediction, which can model the relatedness
between historical behaviors. The second one is next $K$ behavior prediction,
which can model the relatedness between past and future behaviors. The
pre-trained user models are finetuned in downstream tasks to learn
task-specific user representations. Experimental results on two real-world
datasets validate the effectiveness of our proposed user model pre-training
method.

本文提出了一种基于自监督学习的用户模型预训练方法，包括屏蔽行为预测和 $K$ 步预测两个任务，通过对大规模未标注用户行为数据进行预训练，并在下游任务中微调，获得了更好的效果。