Recently, multi-modal content generation has attracted lots of attention from
researchers by investigating the utilization of visual instruction tuning based
on large language models (LLMs). To enhance the performance and generalization
ability of such LLMs, the practice of distilling knowledge from pretrained
multi-modal models (a.k.a. teachers) to more compact multi-modal LLMs
(students) has gained considerable interest. However, the prevailing paradigm
of instructiontuning in multi-modal LLMs knowledge distillation is
resource-intensive and unidirectional, neglecting the potential for mutual
feedback between the student and teacher models. Thus, we propose an innovative
Competitive Multi-modal Distillation framework (CoMD), which captures
bidirectional feedback between teacher and student models and continually
updates the multi-modal capabilities that the student model has learned. It
comprises two stages: multi-modal pre-training and multi-modal competitive
distillation. The first stage pre-trains the student model on a large number of
filtered multi-modal datasets. The second stage facilitates a bidirectional
knowledge transfer between the student and teacher models. Our experimental
analysis of diverse datasets shows that our knowledge transfer method
consistently improves the capabilities of the student model. Finally, the
7B-sized student model after four distillations surpassed the current
state-of-the-art model LLaVA-13B on the ScienceQA and LLaVA Test dataset, also
outperforms other strong baselines in the zero-shot setting.

提出了一种创新的 Competitive Multi-modal Distillation（CoMD）框架，捕捉学生模型和教师模型之间的双向反馈，并不断更新学生模型学到的多模态能力，研究结果表明，我们的知识迁移方法在各种数据集上稳定提升了学生模型的能力，经过四次蒸馏后，7B 大小的学生模型在 ScienceQA 和 LLaVA Test 数据集上超过了当前最先进的 LLaVA-13B 模型，在零样本设置下也优于其他强基线模型。