In this paper, we investigate the length-extension of state-space models
(SSMs) in language modeling. Length extension involves training models on short
sequences and testing them on longer ones. We show that state-space models
trained with zero hidden states initialization have difficulty doing length
extension. We explain this difficulty by pointing out the length extension is
equivalent to polynomial extrapolation. Based on the theory, we propose a
simple yet effective method - changing the hidden states initialization scheme
- to improve the length extension. Moreover, our method shows that using long
training sequence length is beneficial but not necessary to length extension.
Changing the hidden state initialization enables the efficient training of
long-memory model with a smaller training context length.

在这篇论文中，我们研究了语言建模中状态空间模型（SSMs）的长度扩展问题。我们发现，使用零隐藏状态初始化的状态空间模型在长度扩展上存在困难，并将此困难解释为多项式外推的等价形式。基于这个理论，我们提出了一种简单而有效的方法 —— 改变隐藏状态初始化方案 —— 以改进长度扩展。此外，我们的方法表明，使用较长的训练序列长度对于长度扩展是有益的，但并非必要条件。改变隐藏状态初始化使得能够以更小的训练上下文长度有效地训练具有长记忆的模型。