General-purpose representation learning through large-scale pre-training has
shown promising results in the various machine learning fields. For an
e-commerce domain, the objective of general-purpose, i.e., one for all,
representations would be efficient applications for extensive downstream tasks
such as user profiling, targeting, and recommendation tasks. In this paper, we
systematically compare the generalizability of two learning strategies, i.e.,
transfer learning through the proposed model, ShopperBERT, vs. learning from
scratch. ShopperBERT learns nine pretext tasks with 79.2M parameters from 0.8B
user behaviors collected over two years to produce user embeddings. As a
result, the MLPs that employ our embedding method outperform more complex
models trained from scratch for five out of six tasks. Specifically, the
pre-trained embeddings have superiority over the task-specific supervised
features and the strong baselines, which learn the auxiliary dataset for the
cold-start problem. We also show the computational efficiency and embedding
visualization of the pre-trained features.

本研究通过构建模型 ShopperBERT 对九个预训练任务训练得出的预训练嵌入进行广泛的下游任务如用户画像、定向和推荐等方面的有效应用，并与从零开始的学习策略进行了比较，结果表明前者的优势明显，尤其针对冷启动问题。