Multimodal large language models (MLLMs) have attracted widespread interest
and have rich applications. However, the inherent attention mechanism in its
Transformer structure requires quadratic complexity and results in expensive
computational overhead. Therefore, in this work, we propose VL-Mamba, a
multimodal large language model based on state space models, which have been
shown to have great potential for long-sequence modeling with fast inference
and linear scaling in sequence length. Specifically, we first replace the
transformer-based backbone language model such as LLama or Vicuna with the
pre-trained Mamba language model. Then, we empirically explore how to
effectively apply the 2D vision selective scan mechanism for multimodal
learning and the combinations of different vision encoders and variants of
pretrained Mamba language models. The extensive experiments on diverse
multimodal benchmarks with competitive performance show the effectiveness of
our proposed VL-Mamba and demonstrate the great potential of applying state
space models for multimodal learning tasks.

通过基于状态空间模型的 VL-Mamba 多模态大语言模型和 2D 视觉选择扫描机制以及不同视觉编码器和预训练 Mamba 语言模型的组合的实证研究，我们证明了状态空间模型在多模态学习任务中具有巨大潜力，并展示了 VL-Mamba 在各种多模态基准测试中具有竞争力的性能。

VL-Mamba: 多模式学习中的状态空间模型探索

VL-Mamba: Exploring State Space Models for Multimodal Learning

The trade-off between performance and computational efficiency in
long-sequence modeling becomes a bottleneck for existing models. Inspired by
the continuous state space models (SSMs) with multi-input and multi-output in
control theory, we propose a new neural network called Linear Dynamics-embedded
Neural Network (LDNN). SSMs' continuous, discrete, and convolutional properties
enable LDNN to have few parameters, flexible inference, and efficient training
in long-sequence tasks. Two efficient strategies, diagonalization and
$'\text{Disentanglement then Fast Fourier Transform (FFT)}'$, are developed to
reduce the time complexity of convolution from $O(LNH\max\{L, N\})$ to
$O(LN\max \{H, \log L\})$. We further improve LDNN through bidirectional
noncausal and multi-head settings to accommodate a broader range of
applications. Extensive experiments on the Long Range Arena (LRA) demonstrate
the effectiveness and state-of-the-art performance of LDNN.

长序列建模中性能和计算效率之间的权衡成为现有模型的瓶颈，本文基于控制理论中具有多输入和多输出的连续状态空间模型（SSMs），提出了一种新型神经网络，称为线性动力学嵌入神经网络（LDNN）。通过对角化和解耦然后快速傅里叶变换（FFT）的两种高效策略，将卷积的时间复杂度从 O (LNHmax {L, N}) 降低到 O (LNmax {H, log L})，实现了 LDNN 在长序列任务中的少参数、灵活推断和高效训练。我们通过双向非因果和多头设置进一步改进了 LDNN，以适应更广泛的应用领域。对长距离竞技场（LRA）进行了大量实验证明了 LDNN 的有效性和最先进的性能。

线性动力学嵌入神经网络用于长序列建模

Linear Dynamics-embedded Neural Network for Long-Sequence Modeling

Ever since their conception, Transformers have taken over traditional
sequence models in many tasks, such as NLP, image classification, and
video/audio processing, for their fast training and superior performance. Much
of the merit is attributable to positional encoding and multi-head attention.
However, Transformers fall short in learning long-range dependencies mainly due
to the quadratic complexity scaled with context length, in terms of both time
and space. Consequently, over the past five years, a myriad of methods has been
proposed to make Transformers more efficient. In this work, we first take a
step back, study and compare existing solutions to long-sequence modeling in
terms of their pure mathematical formulation. Specifically, we summarize them
using a unified template, given their shared nature of token mixing. Through
benchmarks, we then demonstrate that long context length does yield better
performance, albeit application-dependent, and traditional Transformer models
fall short in taking advantage of long-range dependencies. Next, inspired by
emerging sparse models of huge capacity, we propose a machine learning system
for handling million-scale dependencies. As a proof of concept, we evaluate the
performance of one essential component of this system, namely, the distributed
multi-head attention. We show that our algorithm can scale up attention
computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to
vanilla multi-head attention mechanism. We believe this study is an
instrumental step towards modeling million-scale dependencies.

本文探讨了 Transformers 在长序列建模中的应用，并提出了一种处理百万级依赖关系的机器学习系统，其中的分布式多头注意力机制可提高 40 倍的计算效率。