We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
this https URL

我们提出了 SPHINX-X，这是一个建立在 SPHINX 基础上的广泛的多模态大型语言模型系列。通过去除冗余的视觉编码器、利用跳过标记绕过全填充子图像，以及简化多阶段训练为一阶段的所有合一范式，我们改进了 SPHINX 框架的架构和训练效率。为了充分发挥多模态大型语言模型的潜力，我们组装了一个包括语言、视觉和视觉 - 语言任务中公开可用资源的全面多领域和多模态数据集。我们进一步通过我们精心策划的 OCR 密集和一系列标记数据集丰富了这个收集，扩展了多样性和普适性。通过在不同的基础 LLM 上进行训练，包括 TinyLlama1.1B、InternLM2-7B、LLaMA2-13B 和 Mixtral8x7B，我们获得了一系列参数大小和多语言能力各不相同的 MLLM。全面的基准测试显示，多模态性能与数据和参数规模之间存在着强烈的相关性。代码和模型已在此 URL 发布。