With the widespread application of personalized online services,
click-through rate (CTR) prediction has received more and more attention and
research. The most prominent features of CTR prediction are its multi-field
categorical data format, and vast and daily-growing data volume. The large
capacity of neural models helps digest such massive amounts of data under the
supervised learning paradigm, yet they fail to utilize the substantial data to
its full potential, since the 1-bit click signal is not sufficient to guide the
model to learn capable representations of features and instances. The
self-supervised learning paradigm provides a more promising pretrain-finetune
solution to better exploit the large amount of user click logs, and learn more
generalized and effective representations. However, self-supervised learning
for CTR prediction is still an open question, since current works on this line
are only preliminary and rudimentary. To this end, we propose a Model-agnostic
pretraining (MAP) framework that applies feature corruption and recovery on
multi-field categorical data, and more specifically, we derive two practical
algorithms: masked feature prediction (MFP) and replaced feature detection
(RFD). MFP digs into feature interactions within each instance through masking
and predicting a small portion of input features, and introduces noise
contrastive estimation (NCE) to handle large feature spaces. RFD further turns
MFP into a binary classification mode through replacing and detecting changes
in input features, making it even simpler and more effective for CTR
pretraining. Our extensive experiments on two real-world large-scale datasets
(i.e., Avazu, Criteo) demonstrate the advantages of these two methods on
several strong backbones (e.g., DCNv2, DeepFM), and achieve new
state-of-the-art performance in terms of both effectiveness and efficiency for
CTR prediction.

基于大规模用户点击日志的自我监督学习对于点击率（CTR）预测仍然是一个开放性问题，而我们提出了一个无模型的预训练框架，利用特征破坏和恢复，挖掘多领域分类数据的特征交互，并通过两种实用的算法（即遮蔽特征预测和更替特征检测）进一步优化 CTR 预训练，实现了在 CTR 预测方面的有效性和效率的新的最先进表现。