LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.

LLaVA-Plus 是一个通用的多模态助手，扩展了大型多模态模型的功能，它通过在用户输入的基础上激活相关工具，并维护了一个预训练的视觉和视觉语言模型的技能存储库，以实现实际任务。实证结果表明，LLaVA-Plus 在现有功能上优于 LLaVA，并展示出新的功能。它在直接基于图像查询和整个人工智能与人类交互过程中积极参与中具有独特之处，显著提高了工具使用性能并实现了新的场景。