User modeling is critical for personalized web applications. Existing user
modeling methods usually train user models from user behaviors with
task-specific labeled data. However, labeled data in a target task may be
insufficient for training accurate user models. Fortunately, there are usually
rich unlabeled user behavior data which encode rich information of user
characteristics and interests. Thus, pre-training user models on unlabeled user
behavior data has the potential to improve user modeling for many downstream
tasks. In this paper, we propose a contrastive user model pre-training method
named UserBERT. Two self-supervision tasks are incorporated in UserBERT for
user model pre-training on unlabeled user behavior data to empower user
modeling. The first one is masked behavior prediction, which aims to model the
relatedness between user behaviors. The second one is behavior sequence
matching, which aims to capture the inherent user interests that are consistent
in different periods. In addition, we propose a medium-hard negative sampling
framework to select informative negative samples for better contrastive
pre-training. We maintain a synchronously updated candidate behavior pool and
an asynchronously updated candidate behavior sequence pool to select the
locally hardest negative behaviors and behavior sequences in an efficient way.
Extensive experiments on two real-world datasets in different tasks show that
UserBERT can effectively improve various user models.

本文提出了一种名为 UserBERT 的对比式用户模型预训练方法，将两种自监督任务应用于未标记的用户行为数据的用户模型预训练中，并使用中等难度的负采样框架以更好地进行对比式预训练。实验结果表明，UserBERT 能有效地改善各种用户模型。