Self-supervised learning has been actively studied in time series domain
recently, especially for masked reconstruction. Most of these methods follow
the "Pre-training + Fine-tuning" paradigm in which a new decoder replaces the
pre-trained decoder to fit for a specific downstream task, leading to
inconsistency of upstream and downstream tasks. In this paper, we first point
out that the unification of task objectives and adaptation for task difficulty
are critical for bridging the gap between time series masked reconstruction and
forecasting. By reserving the pre-trained mask token during fine-tuning stage,
the forecasting task can be taken as a special case of masked reconstruction,
where the future values are masked and reconstructed based on history values.
It guarantees the consistency of task objectives but there is still a gap in
task difficulty. Because masked reconstruction can utilize contextual
information while forecasting can only use historical information to
reconstruct. To further mitigate the existed gap, we propose a simple yet
effective prompt token tuning (PT-Tuning) paradigm, in which all pre-trained
parameters are frozen and only a few trainable prompt tokens are added to
extended mask tokens in element-wise manner. Extensive experiments on
real-world datasets demonstrate the superiority of our proposed paradigm with
state-of-the-art performance compared to representation learning and end-to-end
supervised forecasting methods.

本论文针对时间序列遮蔽重构和预测之间的差距，提出了一种任务目标统一和适应任务难度的方法并且通过引入 PT-Tuning 范式进一步减小了差距，实验证明该方法在实际数据集上具有卓越的性能。

PT-Tuning：通过提示词符调优来弥合时间序列掩码重构与预测之间的差距

PT-Tuning: Bridging the Gap between Time Series Masked Reconstruction  and Forecasting via Prompt Token Tuning

Self-supervised learning for computer vision has achieved tremendous progress
and improved many downstream vision tasks such as image classification,
semantic segmentation, and object detection. Among these, generative
self-supervised vision learning approaches such as MAE and BEiT show promising
performance. However, their global masked reconstruction mechanism is
computationally demanding. To address this issue, we propose local masked
reconstruction (LoMaR), a simple yet effective approach that performs masked
reconstruction within a small window of 7$\times$7 patches on a simple
Transformer encoder, improving the trade-off between efficiency and accuracy
compared to global masked reconstruction over the entire image. Extensive
experiments show that LoMaR reaches 84.1% top-1 accuracy on ImageNet-1K
classification, outperforming MAE by 0.5%. After finetuning the pretrained
LoMaR on 384$\times$384 images, it can reach 85.4% top-1 accuracy, surpassing
MAE by 0.6%. On MS COCO, LoMaR outperforms MAE by 0.5 $\text{AP}^\text{box}$ on
object detection and 0.5 $\text{AP}^\text{mask}$ on instance segmentation.
LoMaR is especially more computation-efficient on pretraining high-resolution
images, e.g., it is 3.1$\times$ faster than MAE with 0.2% higher classification
accuracy on pretraining 448$\times$448 images. This local masked reconstruction
learning mechanism can be easily integrated into any other generative
self-supervised learning approach. Our code is publicly available in
this https URL

本研究提出一种名为 LoMaR 的局部掩码重建的学习机制，该方法与生成式自监督视觉学习方法（如 MAE 和 BEiT）相比，可以更加高效地在简单的转换器编码器上进行掩码重建，同时也更具准确性，可轻松集成到其他生成式自监督学习方法中。

具有局部掩码重建的高效自监督视觉预训练

Efficient Self-supervised Vision Pretraining with Local Masked  Reconstruction

Music annotation has always been one of the critical topics in the field of
Music Information Retrieval (MIR). Traditional models use supervised learning
for music annotation tasks. However, as supervised machine learning approaches
increase in complexity, the increasing need for more annotated training data
can often not be matched with available data. In this paper, a new
self-supervised music acoustic representation learning approach named MusiCoder
is proposed. Inspired by the success of BERT, MusiCoder builds upon the
architecture of self-attention bidirectional transformers. Two pre-training
objectives, including Contiguous Frames Masking (CFM) and Contiguous Channels
Masking (CCM), are designed to adapt BERT-like masked reconstruction
pre-training to continuous acoustic frame domain. The performance of MusiCoder
is evaluated in two downstream music annotation tasks. The results show that
MusiCoder outperforms the state-of-the-art models in both music genre
classification and auto-tagging tasks. The effectiveness of MusiCoder indicates
a great potential of a new self-supervised learning approach to understand
music: first apply masked reconstruction tasks to pre-train a transformer-based
model with massive unlabeled music acoustic data, and then finetune the model
on specific downstream tasks with labeled data.

本文提出了一种名为 MusiCoder 的自监督音乐声学表示学习方法，通过掩蔽重建预训练，自适应 BERT 式的自注意力双向转换器架构，该方法在两个下游音乐注释任务中均优于现有模型，表明了自监督学习方法理解音乐的巨大潜力。