Current Multimodal Large Language Models (MLLMs) typically integrate a
pre-trained LLM with another pre-trained vision transformer through a
connector, such as an MLP, endowing the LLM with visual capabilities. However,
the misalignment between two embedding strategies in MLLMs -- the structural
textual embeddings based on an embedding look-up table and the continuous
embeddings generated directly by the vision encoder -- makes challenges for a
more seamless fusion of visual and textual information. We propose Ovis, a
novel MLLM architecture designed to structurally align visual and textual
embeddings. Ovis integrates an additional learnable visual embedding table into
the visual encoder's process. To capture rich visual semantics, each image
patch indexes the visual embedding table multiple times, resulting in a final
visual embedding that is a probabilistic combination of the indexed embeddings.
This structural approach mirrors the method used for generating textual
embeddings. Empirical evaluations on various multimodal benchmarks demonstrate
that Ovis outperforms open-source MLLMs of similar parameter scales and even
surpasses the proprietary model Qwen-VL-Plus overall. These results highlight
the potential of Ovis' structured visual representation for advancing MLLM
architectural design and promoting more effective multimodal learning. Both the
source code and the training dataset of Ovis will be made publicly available.

通过结构对齐视觉和文本嵌入，提出了一种名为 Ovis 的新的多模态大型语言模型体系结构，以捕捉图像的丰富语义，并在各种多模态基准测试中表现出优越性能，从而推动了多模态学习的更有效进展。

Ovis: 多模型大型语言模型的结构嵌入对齐

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

The handling of long videos with complex and occluded sequences has recently
emerged as a new challenge in the video instance segmentation (VIS) community.
However, existing methods have limitations in addressing this challenge. We
argue that the biggest bottleneck in current approaches is the discrepancy
between training and inference. To effectively bridge this gap, we propose a
Generalized framework for VIS, namely GenVIS, that achieves state-of-the-art
performance on challenging benchmarks without designing complicated
architectures or requiring extra post-processing. The key contribution of
GenVIS is the learning strategy, which includes a query-based training pipeline
for sequential learning with a novel target label assignment. Additionally, we
introduce a memory that effectively acquires information from previous states.
Thanks to the new perspective, which focuses on building relationships between
separate frames or clips, GenVIS can be flexibly executed in both online and
semi-online manner. We evaluate our approach on popular VIS benchmarks,
achieving state-of-the-art results on YouTube-VIS 2019/2021/2022 and Occluded
VIS (OVIS). Notably, we greatly outperform the state-of-the-art on the long VIS
benchmark (OVIS), improving 5.6 AP with ResNet-50 backbone. Code is available
at this https URL

提出了基于查询式训练和具有前状态信息的内存模块用于序列学习的通用可变的视频实例分割（GenVIS）方法，并在多个基准测试中取得了优秀的结果。