Recent advancements in Large Multimodal Models (LMMs) have leveraged
extensive multimodal datasets to enhance capabilities in complex
knowledge-driven tasks. However, persistent challenges in perceptual and
reasoning errors limit their efficacy, particularly in interpreting intricate
visual data and deducing multimodal relationships. Addressing these issues, we
introduce a novel dataset format, PIN (Paired and INterleaved multimodal
documents), designed to significantly improve both the depth and breadth of
multimodal training. The PIN format is built on three foundational principles:
knowledge intensity, scalability, and support for diverse training modalities.
This innovative format combines markdown files and comprehensive images to
enrich training data with a dense knowledge structure and versatile training
strategies. We present PIN-14M, an open-source dataset comprising 14 million
samples derived from a diverse range of Chinese and English sources, tailored
to include complex web and scientific content. This dataset is constructed
meticulously to ensure data quality and ethical integrity, aiming to facilitate
advanced training strategies and improve model robustness against common
multimodal training pitfalls. Our initial results, forming the basis of this
technical report, suggest significant potential for the PIN format in refining
LMM performance, with plans for future expansions and detailed evaluations of
its impact on model capabilities.

最近，对大型多模态模型（LMMs）的最新进展借鉴了丰富的多模态数据集，以增强复杂的知识驱动任务的能力。然而，感知和推理错误的持续挑战限制了它们的效力，尤其是在解释复杂视觉数据和推断多模态关系方面。针对这些问题，我们引入了一种新颖的数据集格式，PIN（配对和交错多模态文档），旨在显著提高多模态训练的深度和广度。PIN 格式基于三个基本原则：知识密度、可伸缩性和对不同训练模态的支持。这种创新的格式将 Markdown 文件和全面的图像结合起来，用密集的知识结构和灵活的训练策略丰富训练数据。我们提供了 PIN-14M，一个由来自各种中英文来源的 1400 万个样本构成的开源数据集，旨在包含复杂的网络和科学内容。这个数据集精心构建，以确保数据质量和道德完整性，旨在促进先进的训练策略，并提高模型在常见多模态训练陷阱中的鲁棒性。我们的初步结果成为本技术报告的基础，表明 PIN 格式在提高 LMM 性能方面具有重要潜力，未来还计划扩展并详细评估其对模型能力的影响。