Large Language Models (LLMs), benefiting from the auto-regressive modelling
approach performed on massive unannotated texts corpora, demonstrates powerful
perceptual and reasoning capabilities. However, as for extending
auto-regressive modelling to multi-modal scenarios to build Large Multi-modal
Models (LMMs), there lies a great difficulty that the image information is
processed in the LMM as continuous visual embeddings, which cannot obtain
discrete supervised labels for classification. In this paper, we successfully
perform multi-modal auto-regressive modeling with a unified objective for the
first time. Specifically, we propose the concept of visual words, which maps
the visual features to probability distributions over LLM's vocabulary,
providing supervision information for visual modelling. We further explore the
distribution of visual features in the semantic space within LMM and the
possibility of using text embeddings to represent visual information.
Experimental results and ablation studies on 5 VQA tasks and 4 benchmark
toolkits validate the powerful performance of our proposed approach.

成功进行多模态自回归建模，并首次提出了视觉词概念，将视觉特征映射到 LLMs 词汇的概率分布，为视觉建模提供了监督信息。通过对 5 个 VQA 任务和 4 个基准工具包的实验结果和消融研究的验证，证明了我们提出方法的强大性能。

多模态自回归建模基于视觉单词

Multi-modal Auto-regressive Modeling via Visual Words

Video captioning has been attracting broad research attention in multimedia
community. However, most existing approaches either ignore temporal information
among video frames or just employ local contextual temporal knowledge. In this
work, we propose a novel video captioning framework, termed as
\emph{Bidirectional Long-Short Term Memory} (BiLSTM), which deeply captures
bidirectional global temporal structure in video. Specifically, we first devise
a joint visual modelling approach to encode video data by combining a forward
LSTM pass, a backward LSTM pass, together with visual features from
Convolutional Neural Networks (CNNs). Then, we inject the derived video
representation into the subsequent language model for initialization. The
benefits are in two folds: 1) comprehensively preserving sequential and visual
information; and 2) adaptively learning dense visual features and sparse
semantic representations for videos and sentences, respectively. We verify the
effectiveness of our proposed video captioning framework on a commonly-used
benchmark, i.e., Microsoft Video Description (MSVD) corpus, and the
experimental results demonstrate that the superiority of the proposed approach
as compared to several state-of-the-art methods.

提出一种新型的视频字幕生成框架 Bidirectional Long-Short Term Memory，该框架综合保留了视频的信息，并在常用基准测试上验证了该框架的有效性。