Data curation is an essential component of large-scale pretraining. In this
work, we demonstrate that jointly selecting batches of data is more effective
for learning than selecting examples independently. Multimodal contrastive
objectives expose the dependencies between data and thus naturally yield
criteria for measuring the joint learnability of a batch. We derive a simple
and tractable algorithm for selecting such batches, which significantly
accelerate training beyond individually-prioritized data points. As performance
improves by selecting from larger super-batches, we also leverage recent
advances in model approximation to reduce the associated computational
overhead. As a result, our approach--multimodal contrastive learning with joint
example selection (JEST)--surpasses state-of-the-art models with up to
13$\times$ fewer iterations and 10$\times$ less computation. Essential to the
performance of JEST is the ability to steer the data selection process towards
the distribution of smaller, well-curated datasets via pretrained reference
models, exposing the level of data curation as a new dimension for neural
scaling laws.

数据策展是大规模预训练的重要组成部分。本文中，我们证明了联合选择数据批次比独立选择示例更有效，并且多模态对比目标揭示了数据之间的依赖关系，从而自然地产生衡量批次联合可学习性的标准。我们推导出一种简单易行的选择这类批次的算法，通过它可以明显提高训练速度，超过目前技术水平的模型，在少达 13 倍的迭代次数和 10 倍的计算量下。JEST 的性能关键是将数据选择过程引导到小型、精心策展的数据集的分布上，通过预训练参考模型，将数据策展水平暴露为神经缩放定律的新维度。