Time series prediction plays a crucial role in various industrial fields. In
recent years, neural networks with a transformer backbone have achieved
remarkable success in many domains, including computer vision and NLP. In time
series analysis domain, some studies have suggested that even the simplest MLP
networks outperform advanced transformer-based networks on time series forecast
tasks. However, we believe these findings indicate there to be low-rank
properties in time series sequences. In this paper, we consider the low-pass
characteristics of transformers and try to incorporate the advantages of MLP.
We adopt skip-layer connections inspired by Unet into traditional transformer
backbone, thus preserving high-frequency context from input to output, namely
U-shaped Transformer. We introduce patch merge and split operation to extract
features with different scales and use larger datasets to fully make use of the
transformer backbone. Our experiments demonstrate that the model performs at an
advanced level across multiple datasets with relatively low cost.

时间序列预测是各个工业领域中的关键任务。近年来，具备变换器骨干的神经网络在计算机视觉和自然语言处理等多个领域中取得了显著成功。在时间序列分析领域，一些研究发现即使是最简单的多层感知机网络在时间序列预测任务上也能胜过先进的基于变换器的网络。然而，我们认为这些发现表明时间序列序列中存在低秩特性。本文中，我们考虑了变换器的低通特性，并试图融入多层感知机的优点。我们在传统的变换器骨干中采用了受 Unet 启发的跳跃连接，从而保留了从输入到输出的高频率上下文，即 U 型变换器。我们引入了补丁合并和分割操作，以提取不同尺度的特征，并利用更大的数据集充分利用变换器骨干。我们的实验表明，该模型在多个数据集上表现出较高水平的性能，并具有相对较低的成本。

U 型 Transformer：在时间序列分析中保留高频上下文

U-shaped Transformer: Retain High Frequency Context in Time Series  Analysis

In this paper, we present an efficient method for storing fine-tuned models
by leveraging the low-rank properties of weight residuals. Our key observation
is that weight residuals in large overparameterized models exhibit even
stronger low-rank characteristics. Based on this insight, we propose Efficient
Residual Encoding (ERE), a novel approach that achieves efficient storage of
fine-tuned model weights by approximating the low-rank weight residuals.
Furthermore, we analyze the robustness of weight residuals and push the limit
of storage efficiency by utilizing additional quantization and layer-wise rank
allocation. Our experimental results demonstrate that our method significantly
reduces memory footprint while preserving performance in various tasks and
modalities. We release our code.

通过权重残差的低秩特性，本文提出了一种高效的存储 fine-tuned 模型的方法，称为 Efficient Residual Encoding（ERE），通过低秩权重残差的逼近来实现 fine-tuned 模型权重的高效存储，并通过使用额外的量化和分层秩分配来分析权重残差的健壮性并推进存储效率上限。实验结果表明，ERE 方法在各种任务和模态下显着减少内存占用并保持性能，代码已发布。