Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
the exploration of Artificial General Intelligence (AGI). The core of MLLMs
lies in their capability to achieve cross-modal alignment. To attain this goal,
current MLLMs typically follow a two-phase training paradigm: the pre-training
phase and the instruction-tuning phase. Despite their success, there are
shortcomings in the modeling of alignment capabilities within these models.
Firstly, during the pre-training phase, the model usually assumes that all
image-text pairs are uniformly aligned, but in fact the degree of alignment
between different image-text pairs is inconsistent. Secondly, the instructions
currently used for finetuning incorporate a variety of tasks, different tasks's
instructions usually require different levels of alignment capabilities, but
previous MLLMs overlook these differentiated alignment needs. To tackle these
issues, we propose a new multimodal large language model AlignGPT. In the
pre-training stage, instead of treating all image-text pairs equally, we assign
different levels of alignment capabilities to different image-text pairs. Then,
in the instruction-tuning phase, we adaptively combine these different levels
of alignment capabilities to meet the dynamic alignment needs of different
instructions. Extensive experimental results show that our model achieves
competitive performance on 12 benchmarks.

利用新的多模态大型语言模型 AlignGPT，通过在预训练阶段为不同的图像 - 文本对分配不同级别的对齐能力，并在指导微调阶段自适应地组合这些不同级别的对齐能力，以满足不同指令的动态对齐需求，取得了 12 个基准测试的竞争性性能。

AlignGPT: 具有自适应对齐能力的多模态大型语言模型

AlignGPT: Multi-modal Large Language Models with Adaptive Alignment  Capability

This work introduces LAB (Large-scale Alignment for chatBots), a novel
methodology designed to overcome the scalability challenges in the
instruction-tuning phase of large language model (LLM) training. Leveraging a
taxonomy-guided synthetic data generation process and a multi-phase tuning
framework, LAB significantly reduces reliance on expensive human annotations
and proprietary models like GPT-4. We demonstrate that LAB-trained models can
achieve competitive performance across several benchmarks compared to models
trained with traditional human-annotated or GPT-4 generated synthetic data.
Thus offering a scalable, cost-effective solution for enhancing LLM
capabilities and instruction-following behaviors without the drawbacks of
catastrophic forgetting, marking a step forward in the efficient training of
LLMs for a wide range of applications.

使用 LAB（大规模对话机器人对齐）方法解决大型语言模型（LLM）训练中的可扩展性挑战，通过基于分类法的合成数据生成和多阶段调整框架，减少人工标注和专有模型的依赖，LAB 训练的模型在多个基准测试上表现出与传统人工标注或 GPT-4 生成的合成数据训练模型相媲美的性能，为提高 LLM 能力和指令遵循行为提供了可扩展、经济高效的解决方案，避免了灾难性遗忘的弊端，为各种应用的 LLM 的高效训练迈出了一步。