Advancements in self-supervised pre-training (SSL) have significantly
advanced the field of learning transferable time series representations, which
can be very useful in enhancing the downstream task. Despite being effective,
most existing works struggle to achieve cross-domain SSL pre-training, missing
valuable opportunities to integrate patterns and features from different
domains. The main challenge lies in the significant differences in the
characteristics of time-series data across different domains, such as
variations in the number of channels and temporal resolution scales. To address
this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning
framework to learn transferable knowledge from various domains to largely
benefit the target downstream task. One of the key characteristics of
CrossTimeNet is the newly designed time series tokenization module, which could
effectively convert the raw time series into a sequence of discrete tokens
based on a reconstruction optimization process. Besides, we highlight that
predicting a high proportion of corrupted tokens can be very helpful for
extracting informative patterns across different domains during SSL
pre-training, which has been largely overlooked in past years. Furthermore,
unlike previous works, our work treats the pre-training language model (PLM) as
the initialization of the encoder network, investigating the feasibility of
transferring the knowledge learned by the PLM to the time series area. Through
these efforts, the path to cross-domain pre-training of a generic time series
model can be effectively paved. We conduct extensive experiments in a
real-world scenario across various time series classification domains. The
experimental results clearly confirm CrossTimeNet's superior performance.

通过 CrossTimeNet 提出一种新型的跨领域自监督学习框架，利用时间序列分词模块，从各个领域学习可转移的知识，并且以预训练语言模型作为编码器网络的初始化，取得优于以往工作的卓越性能。

跨领域预训练的可迁移时间序列分类器学习

Learning Transferable Time Series Classifier with Cross-Domain  Pre-training from Language Model

Task-agnostic cross-domain pre-training shows great potential in image-based
Reinforcement Learning (RL) but poses a big challenge. In this paper, we
propose CRPTpro, a Cross-domain self-supervised Random Pre-Training framework
with prototypes for image-based RL. CRPTpro employs cross-domain random policy
to easily and quickly sample diverse data from multiple domains, to improve
pre-training efficiency. Moreover, prototypical representation learning with a
novel intrinsic loss is proposed to pre-train an effective and generic encoder
across different domains. Without finetuning, the cross-domain encoder can be
implemented for challenging downstream visual-control RL tasks defined in
different domains efficiently. Compared with prior arts like APT and Proto-RL,
CRPTpro achieves better performance on cross-domain downstream RL tasks without
extra training on exploration agents for expert data collection, greatly
reducing the burden of pre-training. Experiments on DeepMind Control suite
(DMControl) demonstrate that CRPTpro outperforms APT significantly on 11/12
cross-domain RL tasks with only 39% pre-training hours, becoming a
state-of-the-art cross-domain pre-training method in both policy learning
performance and pre-training efficiency. The complete code will be released at
this https URL.

本文提出了一种基于 CRPTpro 的跨域自监督随机预训练框架，它采用了跨域随机策略和原型表示学习方法，在不同的任务领域中能够更高效地实现图像相关的强化学习任务。CRPTpro 在 DeepMind Control suite (DMControl) 的实验中表现出比 APT 更好的性能，并且能够在更少的预训练时间内实现更高效的预训练。