The remarkable success of modern machine learning models on large datasets
often demands extensive training time and resource consumption. To save cost, a
prevalent research line, known as online batch selection, explores selecting
informative subsets during the training process. Although recent efforts
achieve advancements by measuring the impact of each sample on generalization,
their reliance on additional reference models inherently limits their practical
applications, when there are no such ideal models available. On the other hand,
the vanilla reference-model-free methods involve independently scoring and
selecting data in a sample-wise manner, which sacrifices the diversity and
induces the redundancy. To tackle this dilemma, we propose Diversified Batch
Selection (DivBS), which is reference-model-free and can efficiently select
diverse and representative samples. Specifically, we define a novel selection
objective that measures the group-wise orthogonalized representativeness to
combat the redundancy issue of previous sample-wise criteria, and provide a
principled selection-efficient realization. Extensive experiments across
various tasks demonstrate the significant superiority of DivBS in the
performance-speedup trade-off. The code is publicly available.

现代机器学习模型对大型数据集的卓越成功往往需要大量的训练时间和资源消耗，为了节省成本，一种被广泛研究的方法是在线批次选择，在训练过程中选择信息量较大的子集。然而，现有的方法依赖于额外的参考模型，限制了它们的实际应用，因为这样理想的模型并不总是可用。为了解决这一困境，我们提出了一种无需参考模型的多样化批次选择算法（DivBS），可以有效地选择多样化和代表性的样本。我们定义了一种新的选择目标，以组间正交的代表性来衡量样本冗余问题，并提供了一个有原则的高效选择实现。通过各种任务的广泛实验证明了 DivBS 在性能和加速度之间的显着优势。代码公开可用。