Time series pre-training has recently garnered wide attention for its
potential to reduce labeling expenses and benefit various downstream tasks.
Prior methods are mainly based on pre-training techniques well-acknowledged in
vision or language, such as masked modeling and contrastive learning. However,
randomly masking time series or calculating series-wise similarity will distort
or neglect inherent temporal correlations crucial in time series data. To
emphasize temporal correlation modeling, this paper proposes TimeSiam as a
simple but effective self-supervised pre-training framework for Time series
based on Siamese networks. Concretely, TimeSiam pre-trains Siamese encoders to
capture intrinsic temporal correlations between randomly sampled past and
current subseries. With a simple data augmentation method (e.g.~masking),
TimeSiam can benefit from diverse augmented subseries and learn internal
time-dependent representations through a past-to-current reconstruction.
Moreover, learnable lineage embeddings are also introduced to distinguish
temporal distance between sampled series and further foster the learning of
diverse temporal correlations. TimeSiam consistently outperforms extensive
advanced pre-training baselines, demonstrating superior forecasting and
classification capabilities across 13 standard benchmarks in both intra- and
cross-domain scenarios.

通过引入 TimeSiam 框架，基于 Siamese 网络的时间序列自监督预训练方法能够强调时间相关性建模，通过过去到当前的重构学习内部时态相关的表示，以及引入可学习的线性嵌入来区分采样序列的时间间隔，进而在 13 个标准基准测试中，在跨领域和同领域情境下展示出优越的预测和分类能力。

TimeSiam：Siamese 时间序列建模的预训练框架

TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling

A foundation model is a machine learning model trained on a large and diverse
set of data, typically using self-supervised learning-based pre-training
techniques, that can be adapted to various downstream tasks. However, current
research on time series pre-training has mostly focused on models pre-trained
solely on data from a single domain, resulting in a lack of knowledge about
other types of time series. However, current research on time series
pre-training has predominantly focused on models trained exclusively on data
from a single domain. As a result, these models possess domain-specific
knowledge that may not be easily transferable to time series from other
domains. In this paper, we aim to develop an effective time series foundation
model by leveraging unlabeled samples from multiple domains. To achieve this,
we repurposed the publicly available UCR Archive and evaluated four existing
self-supervised learning-based pre-training methods, along with a novel method,
on the datasets. We tested these methods using four popular neural network
architectures for time series to understand how the pre-training methods
interact with different network designs. Our experimental results show that
pre-training improves downstream classification tasks by enhancing the
convergence of the fine-tuning process. Furthermore, we found that the proposed
pre-training method, when combined with the Transformer model, outperforms the
alternatives.

本文通过利用多个领域的无标签样本，旨在开发一种有效的时间序列基础模型。实验结果表明，与其他方法相比，该提出的预训练方法结合 Transformer 模型在下游分类任务中具有更好的性能。