Large multimodal models demonstrate remarkable generalist ability to perform
diverse multimodal tasks in a zero-shot manner. Large-scale web-based
image-text pairs contribute fundamentally to this success, but suffer from
excessive noise. Recent studies use alternative captions synthesized by
captioning models and have achieved notable benchmark performance. However, our
experiments reveal significant Scalability Deficiency and World Knowledge Loss
issues in models trained with synthetic captions, which have been largely
obscured by their initial benchmark success. Upon closer examination, we
identify the root cause as the overly-simplified language structure and lack of
knowledge details in existing synthetic captions. To provide higher-quality and
more scalable multimodal pretraining data, we propose CapsFusion, an advanced
framework that leverages large language models to consolidate and refine
information from both web-based image-text pairs and synthetic captions.
Extensive experiments show that CapsFusion captions exhibit remarkable
all-round superiority over existing captions in terms of model performance
(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
efficiency (requiring 11-16 times less computation than baselines), world
knowledge depth, and scalability. These effectiveness, efficiency and
scalability advantages position CapsFusion as a promising candidate for future
scaling of LMM training.

为了提供更高质量和更可扩展的多模态预训练数据，我们提出了 CapsFusion，一种先进的框架，利用大型语言模型从基于网络的图像文本对和合成字幕中巩固和完善信息。大量实验证明，CapsFusion 字幕在模型性能（例如，COCO 和 NoCaps 上的 CIDEr 分数提高了 18.8％和 18.3％）、样本效率（比基准计算需求少 11-16 倍）、世界知识深度和可扩展性方面展示出卓越的全面优势。这种有效性、效率和可扩展性优势使 CapsFusion 成为未来大规模多模态模型训练的有希望的候选者。