In ImageNet-condensation, the storage for auxiliary soft labels exceeds that of the condensed dataset by over 30 times. However, are large-scale soft labels necessary for large-scale dataset distillation? In this paper, we first discover that the high within-class similarity in condensed datasets necessitates the use of large-scale soft labels. This high within-class similarity can be attributed to the fact that previous methods use samples from different classes to construct a single batch for batch normalization (BN) matching. To reduce the within-class similarity, we introduce class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes. As a result, we can increase within-class diversity and reduce the size of required soft labels. A key benefit of improved image diversity is that soft label compression can be achieved through simple random pruning, eliminating the need for complex rule-based strategies. Experiments validate our discoveries. For example, when condensing ImageNet-1K to 200 images per class, our approach compresses the required soft labels from 113 GB to 2.8 GB (40x compression) with a 2.6% performance gain. Code is available at: https://github.com/he-y/soft-label-pruning-for-dataset-distillation

本研究探讨了大规模软标签在大规模数据集蒸馏中的必要性，重点解决了压缩数据集时类内相似性过高的问题。通过在图像合成过程中引入类级监督，显著提高了类内多样性，从而减少了软标签的需求。研究表明，采用这种方法时，所需软标签的大小可以从113 GB压缩到2.8 GB，同时性能提升了2.6%。

大规模软标签对于大规模数据集蒸馏是否必要？