Foundation models have recently gained attention within the field of machine
learning thanks to its efficiency in broad data processing. While researchers
had attempted to extend this success to time series models, the main challenge
is effectively extracting representations and transferring knowledge from
pretraining datasets to the target finetuning dataset. To tackle this issue, we
introduce a novel pretraining procedure that leverages supervised contrastive
learning to distinguish features within each pretraining dataset. This
pretraining phase enables a probabilistic similarity metric, which assesses the
likelihood of a univariate sample being closely related to one of the
pretraining datasets. Subsequently, using this similarity metric as a guide, we
propose a fine-tuning procedure designed to enhance the accurate prediction of
the target data by aligning it more closely with the learned dynamics of the
pretraining datasets. Our experiments have shown promising results which
demonstrate the efficacy of our approach.

引入一种新的预训练程序，利用有监督对比学习来区分每个预训练数据集中的特征，进而通过将目标数据与预训练数据集的学习动态更加紧密地对齐，以提高目标数据的准确预测。

一种用于时间序列的监督对比学习预训练 - 微调方法

A Supervised Contrastive Learning Pretrain-Finetune Approach for Time  Series

Current state-of-the-art deep networks are all powered by backpropagation. In
this paper, we explore alternatives to full backpropagation in the form of
blockwise learning rules, leveraging the latest developments in self-supervised
learning. We show that a blockwise pretraining procedure consisting of training
independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins'
loss function at each block performs almost as well as end-to-end
backpropagation on ImageNet: a linear probe trained on top of our blockwise
pretrained model obtains a top-1 classification accuracy of 70.48%, only 1.1%
below the accuracy of an end-to-end pretrained network (71.57% accuracy). We
perform extensive experiments to understand the impact of different components
within our method and explore a variety of adaptations of self-supervised
learning to the blockwise paradigm, building an exhaustive understanding of the
critical avenues for scaling local learning rules to large networks, with
implications ranging from hardware design to neuroscience.

该论文中，我们使用最新的自监督学习技术，探索了全面反向传播的替代方法，即以块为单位的学习规则。我们证明了一个块状预训练程序，由 ResNet-50 的 4 个主要层块独立训练并使用 Barlow Twins' loss 函数，在 ImageNet 数据集上表现几乎与端到端反向传播相同，达到了 70.48% 的 top-1 分类准确率。