Self-supervision is one of the hallmarks of representation learning in the
increasingly popular suite of foundation models including large language models
such as BERT and GPT-3, but it has not been pursued in the context of
multivariate event streams, to the best of our knowledge. We introduce a new
paradigm for self-supervised learning for multivariate point processes using a
transformer encoder. Specifically, we design a novel pre-training strategy for
the encoder where we not only mask random event epochs but also insert randomly
sampled "void" epochs where an event does not occur; this differs from the
typical discrete-time pretext tasks such as word-masking in BERT but expands
the effectiveness of masking to better capture continuous-time dynamics. To
improve downstream tasks, we introduce a contrasting module that compares real
events to simulated void instances. The pre-trained model can subsequently be
fine-tuned on a potentially much smaller event dataset, similar conceptually to
the typical transfer of popular pre-trained language models. We demonstrate the
effectiveness of our proposed paradigm on the next-event prediction task using
synthetic datasets and 3 real applications, observing a relative performance
boost of as high as up to 20% compared to state-of-the-art models.

使用 Transformer 编码器的自监督学习方法来处理多元事件流数据，在预训练阶段引入了新的策略，通过随机遮盖事件时刻和插入不存在事件的时刻来扩展遮盖的效果，通过对比真实事件和模拟的不存在事件来提高下游任务的性能，实验证明相较于现有模型，该方法在下一个事件预测任务上能够提升高达 20% 的性能。