Transformer architectures have shown promising results in time series
processing. However, despite recent advances in subquadratic attention
mechanisms or state-space models, processing very long sequences still imposes
significant computational requirements. Token merging, which involves replacing
multiple tokens with a single one calculated as their linear combination, has
shown to considerably improve the throughput of vision transformer
architectures while maintaining accuracy. In this work, we go beyond computer
vision and perform the first investigations of token merging in time series
analysis on both time series transformers and state-space models. To
effectively scale token merging to long sequences, we introduce local merging,
a domain-specific token merging algorithm that selectively combines tokens
within a local neighborhood, adjusting the computational complexity from linear
to quadratic based on the neighborhood size. Our comprehensive empirical
evaluation demonstrates that token merging offers substantial computational
benefits with minimal impact on accuracy across various models and datasets. On
the recently proposed Chronos foundation model, we achieve accelerations up to
5400% with only minor accuracy degradations.

在时间序列分析中，我们首次探讨了令牌合并在时间序列变换器和状态空间模型中的应用，通过引入局部合并算法，我们能够在长序列上有效实现令牌合并，从而在多个模型和数据集上获得显著的计算效益而对准确性影响最小。

基于标记合并的转换器和状态空间模型的高效时间序列处理

Efficient Time Series Processing for Transformers and State-Space Models  through Token Merging

Markov jump processes and continuous time Bayesian networks are important
classes of continuous time dynamical systems. In this paper, we tackle the
problem of inferring unobserved paths in these models by introducing a fast
auxiliary variable Gibbs sampler. Our approach is based on the idea of
uniformization, and sets up a Markov chain over paths by sampling a finite set
of virtual jump times and then running a standard hidden Markov model forward
filtering-backward sampling algorithm over states at the set of extant and
virtual jump times. We demonstrate significant computational benefits over a
state-of-the-art Gibbs sampler on a number of continuous time Bayesian
networks.

本文提出了基于 Gibbs 采样的快速辅助变量算法，利用 uniformization 方法和隐式马尔可夫模型来推断这些模型中的未观测路径，并通过在一系列连续时间贝叶斯网络上展示了比最先进的 Gibbs 采样算法更显著的计算优势。