Current Multimodal Large Language Models (MLLMs) typically integrate a
pre-trained LLM with another pre-trained vision transformer through a
connector, such as an MLP, endowing the LLM with visual capabilities. However,
the misalignment between two embedding strategies in MLLMs -- the structural
textual embeddings based on an embedding look-up table and the continuous
embeddings generated directly by the vision encoder -- makes challenges for a
more seamless fusion of visual and textual information. We propose Ovis, a
novel MLLM architecture designed to structurally align visual and textual
embeddings. Ovis integrates an additional learnable visual embedding table into
the visual encoder's process. To capture rich visual semantics, each image
patch indexes the visual embedding table multiple times, resulting in a final
visual embedding that is a probabilistic combination of the indexed embeddings.
This structural approach mirrors the method used for generating textual
embeddings. Empirical evaluations on various multimodal benchmarks demonstrate
that Ovis outperforms open-source MLLMs of similar parameter scales and even
surpasses the proprietary model Qwen-VL-Plus overall. These results highlight
the potential of Ovis' structured visual representation for advancing MLLM
architectural design and promoting more effective multimodal learning. Both the
source code and the training dataset of Ovis will be made publicly available.

通过结构对齐视觉和文本嵌入，提出了一种名为 Ovis 的新的多模态大型语言模型体系结构，以捕捉图像的丰富语义，并在各种多模态基准测试中表现出优越性能，从而推动了多模态学习的更有效进展。