Time series forecasting plays a crucial role in decision-making across
various domains, but it presents significant challenges. Recent studies have
explored image-driven approaches using computer vision models to address these
challenges, often employing lineplots as the visual representation of time
series data. In this paper, we propose a novel approach that uses
time-frequency spectrograms as the visual representation of time series data.
We introduce the use of a vision transformer for multimodal learning,
showcasing the advantages of our approach across diverse datasets from
different domains. To evaluate its effectiveness, we compare our method against
statistical baselines (EMA and ARIMA), a state-of-the-art deep learning-based
approach (DeepAR), other visual representations of time series data (lineplot
images), and an ablation study on using only the time series as input. Our
experiments demonstrate the benefits of utilizing spectrograms as a visual
representation for time series data, along with the advantages of employing a
vision transformer for simultaneous learning in both the time and frequency
domains.

通过使用时间频谱图作为时间序列数据的可视表示，使用视觉变换器进行多模态学习的一种新方法，与统计基线、深度学习方法、其他视觉表示方法和只使用时间序列作为输入的研究进行对比，表明利用频谱图作为时间序列数据的可视表示具有优势，并且同时学习时间和频率域的优势。

从像素到预测：频谱图和视觉变换器用于更好的时间序列预测

From Pixels to Predictions: Spectrogram and Vision Transformer for  Better Time Series Forecasting

Low-channel EEG devices are crucial for portable and entertainment
applications. However, the low spatial resolution of EEG presents challenges in
decoding low-channel motor imagery. This study introduces TSFF-Net, a novel
network architecture that integrates time-space-frequency features, effectively
compensating for the limitations of single-mode feature extraction networks
based on time-series or time-frequency modalities. TSFF-Net comprises four main
components: time-frequency representation, time-frequency feature extraction,
time-space feature extraction, and feature fusion and classification.
Time-frequency representation and feature extraction transform raw EEG signals
into time-frequency spectrograms and extract relevant features. The time-space
network processes time-series EEG trials as input and extracts temporal-spatial
features. Feature fusion employs MMD loss to constrain the distribution of
time-frequency and time-space features in the Reproducing Kernel Hilbert Space,
subsequently combining these features using a weighted fusion approach to
obtain effective time-space-frequency features. Moreover, few studies have
explored the decoding of three-channel motor imagery based on time-frequency
spectrograms. This study proposes a shallow, lightweight decoding architecture
(TSFF-img) based on time-frequency spectrograms and compares its classification
performance in low-channel motor imagery with other methods using two publicly
available datasets. Experimental results demonstrate that TSFF-Net not only
compensates for the shortcomings of single-mode feature extraction networks in
EEG decoding, but also outperforms other state-of-the-art methods. Overall,
TSFF-Net offers considerable advantages in decoding low-channel motor imagery
and provides valuable insights for algorithmically enhancing low-channel EEG
decoding.

本研究介绍了一种新的网络架构 TSFF-Net，它集成了时间 - 空间 - 频率特征，有效地弥补了基于时间序列和时频模态的单模特征提取网络的局限性，用于解码三通道运动想象的分类，结果表明，TSFF-Net 在 EEG 解码方面具有显著优势并为算法增强低通道 EEG 解码提供了有价值的洞见。