Transformer-based architectures achieved breakthrough performance in natural
language processing and computer vision, yet they remain inferior to simpler
linear baselines in multivariate long-term forecasting. To better understand
this phenomenon, we start by studying a toy linear forecasting problem for
which we show that transformers are incapable of converging to their true
solution despite their high expressive power. We further identify the attention
of transformers as being responsible for this low generalization capacity.
Building upon this insight, we propose a shallow lightweight transformer model
that successfully escapes bad local minima when optimized with sharpness-aware
optimization. We empirically demonstrate that this result extends to all
commonly used real-world multivariate time series datasets. In particular,
SAMformer surpasses the current state-of-the-art model TSMixer by 14.33% on
average, while having ~4 times fewer parameters. The code is available at
this https URL

通过研究一个玩具线性预测问题，我们发现变压器尽管具有高表达能力，但不能收敛于真实解，这是由于其注意力机制的低泛化能力。基于这一发现，我们提出了一种浅层轻量级变压器模型，在利用尖锐感知优化时能够成功逃离糟糕的局部最小值。我们通过实验证明这一结果在所有常用的多变量时间序列数据集上成立，并且 SAMformer 平均超过当前最先进模型 TSMixer 14.33%，同时参数数量仅为其四分之一。