Learning time-series representations for discriminative tasks has been a
long-standing challenge. Current pre-training methods are limited in either
unidirectional next-token prediction or randomly masked token prediction. We
propose a novel architecture called Bidirectional Timely Generative Pre-trained
Transformer (BiTimelyGPT), which pre-trains on time-series data by both
next-token and previous-token predictions in alternating transformer layers.
This pre-training task preserves original distribution and data shapes of the
time-series. Additionally, the full-rank forward and backward attention
matrices exhibit more expressive representation capabilities. Using biosignal
data, BiTimelyGPT demonstrates superior performance in predicting neurological
functionality, disease diagnosis, and physiological signs. By visualizing the
attention heatmap, we observe that the pre-trained BiTimelyGPT can identify
discriminative segments from time-series sequences, even more so after
fine-tuning on the task.

通过时间序列数据的下一个令牌和上一个令牌的预测，BiTimelyGPT 在预训练任务中保留了时间序列的原始分布和数据形状，并展示了更具表现力的表示能力。使用生物信号数据，BiTimelyGPT 在预测神经功能、疾病诊断和生理迹象方面表现出优异性能。通过可视化注意力热图，我们观察到经过预训练的 BiTimelyGPT 能够从时间序列序列中识别出有区别性的片段，尤其是在任务微调之后。