Visual Language Models (VLMs) have rapidly progressed with the recent success
of large language models. However, there have been few attempts to incorporate
efficient linear Recurrent Neural Networks (RNNs) architectures into VLMs. In
this study, we introduce VisualRWKV, the first application of a linear RNN
model to multimodal learning tasks, leveraging the pre-trained RWKV language
model. We propose a data-dependent recurrence and sandwich prompts to enhance
our modeling capabilities, along with a 2D image scanning mechanism to enrich
the processing of visual sequences. Extensive experiments demonstrate that
VisualRWKV achieves competitive performance compared to Transformer-based
models like LLaVA-1.5 on various benchmarks. To facilitate further research and
analysis, we have made the checkpoints and the associated code publicly
accessible at the following GitHub repository:
\href{https://github.com/howard-hou/VisualRWKV}{this https URL}.

通过引入一种线性循环神经网络模型到多模态学习任务中，该研究提出了 VisualRWKV，它是首个应用线性 RNN 模型到视觉语言模型中的尝试，并通过数据相关的循环和提示增强建模能力，以及 2D 图像扫描机制来丰富对视觉序列的处理，实验证明 VisualRWKV 在各项基准测试中与基于 Transformer 的模型（如 LLaVA-1.5）相比具有竞争性能。

VisualRWKV：探索用于视觉语言模型的递归神经网络

VisualRWKV: Exploring Recurrent Neural Networks for Visual Language  Models

Transformers have surpassed RNNs in popularity due to their superior
abilities in parallel training and long-term dependency modeling. Recently,
there has been a renewed interest in using linear RNNs for efficient sequence
modeling. These linear RNNs often employ gating mechanisms in the output of the
linear recurrence layer while ignoring the significance of using forget gates
within the recurrence. In this paper, we propose a gated linear RNN model
dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes
forget gates that are lower bounded by a learnable value. The lower bound
increases monotonically when moving up layers. This allows the upper layers to
model long-term dependencies and the lower layers to model more local,
short-term dependencies. Experiments on language modeling, image
classification, and long-range arena benchmarks showcase the efficiency and
effectiveness of our proposed model. The source code is available at
this https URL

提出了一种具有遗忘门的分层门控递归神经网络 (HGRN) 模型，其中遗忘门受可学习值下界限制，使得上层能够建模长期依赖，而下层能够建模更局部、短期的依赖关系。通过在语言建模、图像分类和长距离竞技场测试中进行实验，证明了该模型的高效性和有效性。

层次门控循环神经网络用于序列建模

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Contemporary wisdom based on empirical studies suggests that standard
recurrent neural networks (RNNs) do not perform well on tasks requiring
long-term memory. However, precise reasoning for this behavior is still
unknown. This paper provides a rigorous explanation of this property in the
special case of linear RNNs. Although this work is limited to linear RNNs, even
these systems have traditionally been difficult to analyze due to their
non-linear parameterization. Using recently-developed kernel regime analysis,
our main result shows that linear RNNs learned from random initializations are
functionally equivalent to a certain weighted 1D-convolutional network.
Importantly, the weightings in the equivalent model cause an implicit bias to
elements with smaller time lags in the convolution and hence, shorter memory.
The degree of this bias depends on the variance of the transition kernel matrix
at initialization and is related to the classic exploding and vanishing
gradients problem. The theory is validated in both synthetic and real data
experiments.

研究说明传统的循环神经网络（RNNs）在需要长期记忆的任务上表现不佳的原因是因为其随机初始化后的转移矩阵方差造成了梯度消失和梯度爆炸的问题，而使用线性 RNNs 代替时会出现更短的记忆偏差，这一理论经过人工数据和真实数据的验证。