A longstanding challenge for the Machine Learning community is the one of
developing models that are capable of processing and learning from very long
sequences of data. The outstanding results of Transformers-based networks
(e.g., Large Language Models) promotes the idea of parallel attention as the
key to succeed in such a challenge, obfuscating the role of classic sequential
processing of Recurrent Models. However, in the last few years, researchers who
were concerned by the quadratic complexity of self-attention have been
proposing a novel wave of neural models, which gets the best from the two
worlds, i.e., Transformers and Recurrent Nets. Meanwhile, Deep Space-State
Models emerged as robust approaches to function approximation over time, thus
opening a new perspective in learning from sequential data, followed by many
people in the field and exploited to implement a special class of (linear)
Recurrent Neural Networks. This survey is aimed at providing an overview of
these trends framed under the unifying umbrella of Recurrence. Moreover, it
emphasizes novel research opportunities that become prominent when abandoning
the idea of processing long sequences whose length is known-in-advance for the
more realistic setting of potentially infinite-length sequences, thus
intersecting the field of lifelong-online learning from streamed data.

深度学习中基于 Transformer 和循环神经网络的顺序处理对于处理长序列数据和无限长度序列数据具有重要意义。