Advancements in self-supervised pre-training (SSL) have significantly
advanced the field of learning transferable time series representations, which
can be very useful in enhancing the downstream task. Despite being effective,
most existing works struggle to achieve cross-domain SSL pre-training, missing
valuable opportunities to integrate patterns and features from different
domains. The main challenge lies in the significant differences in the
characteristics of time-series data across different domains, such as
variations in the number of channels and temporal resolution scales. To address
this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning
framework to learn transferable knowledge from various domains to largely
benefit the target downstream task. One of the key characteristics of
CrossTimeNet is the newly designed time series tokenization module, which could
effectively convert the raw time series into a sequence of discrete tokens
based on a reconstruction optimization process. Besides, we highlight that
predicting a high proportion of corrupted tokens can be very helpful for
extracting informative patterns across different domains during SSL
pre-training, which has been largely overlooked in past years. Furthermore,
unlike previous works, our work treats the pre-training language model (PLM) as
the initialization of the encoder network, investigating the feasibility of
transferring the knowledge learned by the PLM to the time series area. Through
these efforts, the path to cross-domain pre-training of a generic time series
model can be effectively paved. We conduct extensive experiments in a
real-world scenario across various time series classification domains. The
experimental results clearly confirm CrossTimeNet's superior performance.

通过 CrossTimeNet 提出一种新型的跨领域自监督学习框架，利用时间序列分词模块，从各个领域学习可转移的知识，并且以预训练语言模型作为编码器网络的初始化，取得优于以往工作的卓越性能。