In the 1990s, the constant error carousel and gating were introduced as the
central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have
stood the test of time and contributed to numerous deep learning success
stories, in particular they constituted the first Large Language Models (LLMs).
However, the advent of the Transformer technology with parallelizable
self-attention at its core marked the dawn of a new era, outpacing LSTMs at
scale. We now raise a simple question: How far do we get in language modeling
when scaling LSTMs to billions of parameters, leveraging the latest techniques
from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we
introduce exponential gating with appropriate normalization and stabilization
techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM
with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that
is fully parallelizable with a matrix memory and a covariance update rule.
Integrating these LSTM extensions into residual block backbones yields xLSTM
blocks that are then residually stacked into xLSTM architectures. Exponential
gating and modified memory structures boost xLSTM capabilities to perform
favorably when compared to state-of-the-art Transformers and State Space
Models, both in performance and scaling.

我们修改并扩展 LSTM 的门控机制和记忆结构，得到了 xLSTM 模型，该模型在性能和规模上与最先进的 Transformer 模型和状态空间模型相比表现出色。

xLSTM：扩展的长短期记忆

xLSTM: Extended Long Short-Term Memory

Understanding trajectory diversity is a fundamental aspect of addressing
practical traffic tasks. However, capturing the diversity of trajectories
presents challenges, particularly with traditional machine learning and
recurrent neural networks due to the requirement of large-scale parameters. The
emerging Transformer technology, renowned for its parallel computation
capabilities enabling the utilization of models with hundreds of millions of
parameters, offers a promising solution. In this study, we apply the
Transformer architecture to traffic tasks, aiming to learn the diversity of
trajectories within vehicle populations. We analyze the Transformer's attention
mechanism and its adaptability to the goals of traffic tasks, and subsequently,
design specific pre-training tasks. To achieve this, we create a data structure
tailored to the attention mechanism and introduce a set of noises that
correspond to spatio-temporal demands, which are incorporated into the
structured data during the pre-training process. The designed pre-training
model demonstrates excellent performance in capturing the spatial distribution
of the vehicle population, with no instances of vehicle overlap and an RMSE of
0.6059 when compared to the ground truth values. In the context of time series
prediction, approximately 95% of the predicted trajectories' speeds closely
align with the true speeds, within a deviation of 7.5144m/s. Furthermore, in
the stability test, the model exhibits robustness by continuously predicting a
time series ten times longer than the input sequence, delivering smooth
trajectories and showcasing diverse driving behaviors. The pre-trained model
also provides a good basis for downstream fine-tuning tasks. The number of
parameters of our model is over 50 million.

使用 Transformer 技术，通过分析其注意机制并设计特定的预训练任务，可以学习车辆群体内轨迹的多样性，同时预训练模型在捕捉车辆群体的空间分布方面表现出色，并能准确预测时间序列中的轨迹速度。