A new breed of gated-linear recurrent neural networks has reached
state-of-the-art performance on a range of sequence modeling problems. Such
models naturally handle long sequences efficiently, as the cost of processing a
new input is independent of sequence length. Here, we explore another advantage
of these stateful sequence models, inspired by the success of model merging
through parameter interpolation. Building on parallels between fine-tuning and
in-context learning, we investigate whether we can treat internal states as
task vectors that can be stored, retrieved, and then linearly combined,
exploiting the linearity of recurrence. We study this form of fast model
merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary
evidence that simple linear state interpolation methods suffice to improve
next-token perplexity as well as downstream in-context learning task
performance.

探索利用状态序列模型的内部状态作为任务向量进行快速模型合并的方法，并通过简单的线性状态插值方法提高下一个标记的困惑度和上下文学习任务性能。