Recent advancements in Multimodal Large Language Models (MLLMs) underscore
the significance of scalable models and data to boost performance, yet this
often incurs substantial computational costs. Although the Mixture of Experts
(MoE) architecture has been employed to efficiently scale large language and
image-text models, these efforts typically involve fewer experts and limited
modalities. To address this, our work presents the pioneering attempt to
develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle
a wide array of modalities. Specifically, it features modality-specific
encoders with connectors for a unified multimodal representation. We also
implement a sparse MoE architecture within the LLMs to enable efficient
training and inference through modality-level data parallelism and expert-level
model parallelism. To enhance the multi-expert collaboration and
generalization, we present a progressive training strategy: 1) Cross-modality
alignment using various connectors with different cross-modality data, 2)
Training modality-specific experts with cross-modality instruction data to
activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing
Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate
the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
The extensive experimental results demonstrate Uni-MoE's principal advantage of
significantly reducing performance bias in handling mixed multimodal datasets,
alongside improved multi-expert collaboration and generalization. Our findings
highlight the substantial potential of MoE frameworks in advancing MLLMs and
the code is available at
this https URL

开发一种用于多模态大型语言模型的混合专家 (Uni-MoE) 架构，具有一致的多模态表示，通过稀疏的模态级数据并行和专家级模型并行实现高效的训练和推理，进一步减少性能偏差，并提高多专家的协作和泛化能力。