Recent research on time-series self-supervised models shows great promise in
learning semantic representations. However, it has been limited to small-scale
datasets, e.g., thousands of temporal sequences. In this work, we make key
technical contributions that are tailored to the numerical properties of
time-series data and allow the model to scale to large datasets, e.g., millions
of temporal sequences. We adopt the Transformer architecture by first
partitioning the input into non-overlapping windows. Each window is then
characterized by its normalized shape and two scalar values denoting the mean
and standard deviation within each window. To embed scalar values that may
possess arbitrary numerical scales to high-dimensional vectors, we propose a
numerically multi-scaled embedding module enumerating all possible scales for
the scalar values. The model undergoes pretraining using the proposed
numerically multi-scaled embedding with a simple contrastive objective on a
large-scale dataset containing over a million sequences. We study its transfer
performance on a number of univariate and multivariate classification
benchmarks. Our method exhibits remarkable improvement against previous
representation learning approaches and establishes the new state of the art,
even compared with domain-specific non-learning-based methods.

这篇论文介绍了一种基于时间序列的自监督模型，通过采用 Transformer 架构和多尺度嵌入模块，在大规模数据集上取得了显著的改进，建立了新的最优实验结果。