Recent efforts have been dedicated to enhancing time series forecasting
accuracy by introducing advanced network architectures and self-supervised
pretraining strategies. Nevertheless, existing approaches still exhibit two
critical drawbacks. Firstly, these methods often rely on a single dataset for
training, limiting the model's generalizability due to the restricted scale of
the training data. Secondly, the one-step generation schema is widely followed,
which necessitates a customized forecasting head and overlooks the temporal
dependencies in the output series, and also leads to increased training costs
under different horizon length settings.
To address these issues, we propose a novel generative pretrained
hierarchical transformer architecture for forecasting, named GPHT. There are
two aspects of key designs in GPHT. On the one hand, we advocate for
constructing a mixed dataset for pretraining our model, comprising various
datasets from diverse data scenarios. This approach significantly expands the
scale of training data, allowing our model to uncover commonalities in time
series data and facilitating improved transfer to specific datasets. On the
other hand, GPHT employs an auto-regressive forecasting approach under the
channel-independent assumption, effectively modeling temporal dependencies in
the output series. Importantly, no customized forecasting head is required,
enabling a single model to forecast at arbitrary horizon settings. We conduct
sufficient experiments on eight datasets with mainstream self-supervised
pretraining models and supervised models. The results demonstrated that GPHT
surpasses the baseline models across various fine-tuning and zero/few-shot
learning settings in the traditional long-term forecasting task, providing
support for verifying the feasibility of pretrained time series large models.

提出了一种名为 GPHT 的新型生成预训练分层变压器架构，通过引入高级网络结构和自监督预训练策略来提高时间序列预测的准确性，并在充足的实验验证中证明其超越了传统预训练模型和监督模型，在传统长期预测任务中取得了良好的效果。