In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as multimodal understanding, visual question answering, and video understanding, highlighting its broad application potential.

本文提出了LLaVA-Octopus，一种新型的视频多模态大型语言模型，旨在解决不同视觉投影器在特定任务中的特征权重分配问题。通过根据用户指令动态调整特征权重，LLaVA-Octopus能有效融合各个投影器的优点，显著提升多模态任务的性能，特别是在多模态理解、视觉问答和视频理解等领域展现出广泛的应用潜力。

LLaVA-Octopus：解锁基于指令的自适应投影融合用于视频理解