Learning user sequence behaviour embedding is very sophisticated and
challenging due to the complicated feature interactions over time and high
dimensions of user features. Recent emerging foundation models, e.g., BERT and
its variants, encourage a large body of researchers to investigate in this
field. However, unlike natural language processing (NLP) tasks, the parameters
of user behaviour model come mostly from user embedding layer, which makes most
existing works fail in training a universal user embedding of large scale.
Furthermore, user representations are learned from multiple downstream tasks,
and the past research work do not address the seesaw phenomenon. In this paper,
we propose SUPERMOE, a generic framework to obtain high quality user
representation from multiple tasks. Specifically, the user behaviour sequences
are encoded by MoE transformer, and we can thus increase the model capacity to
billions of parameters, or even to trillions of parameters. In order to deal
with seesaw phenomenon when learning across multiple tasks, we design a new
loss function with task indicators. We perform extensive offline experiments on
public datasets and online experiments on private real-world business
scenarios. Our approach achieves the best performance over state-of-the-art
models, and the results demonstrate the effectiveness of our framework.

本研究提出了一种通用框架，名为 SUPERMOE，用于从多个任务中获得高质量的用户表示，具体方法是通过 MoE 变压器对用户行为序列进行编码，利用任务指标设计了新的损失函数，实验结果表明 OUR 方法在公共数据集和私有真实世界业务场景上取得了最佳性能。