The lightweight "local-match-global" matching introduced by SRe2L
successfully creates a distilled dataset with comprehensive information on the
full 224x224 ImageNet-1k. However, this one-sided approach is limited to a
particular backbone, layer, and statistics, which limits the improvement of the
generalization of a distilled dataset. We suggest that sufficient and various
"local-match-global" matching are more precise and effective than a single one
and has the ability to create a distilled dataset with richer information and
better generalization. We call this perspective "generalized matching" and
propose Generalized Various Backbone and Statistical Matching (G-VBSM) in this
work, which aims to create a synthetic dataset with densities, ensuring
consistency with the complete dataset across various backbones, layers, and
statistics. As experimentally demonstrated, G-VBSM is the first algorithm to
obtain strong performance across both small-scale and large-scale datasets.
Specifically, G-VBSM achieves a performance of 38.7% on CIFAR-100 with
128-width ConvNet, 47.6% on Tiny-ImageNet with ResNet18, and 31.4% on the full
224x224 ImageNet-1k with ResNet18, under images per class (IPC) 10, 50, and 10,
respectively. These results surpass all SOTA methods by margins of 3.9%, 6.5%,
and 10.1%, respectively.

通过引入轻量级的 “局部匹配 - 全局匹配” 机制，利用 Generalized Various Backbone and Statistical Matching（G-VBSM）方法成功创建了一个包含全面信息的精简数据集，该方法比单一匹配更精确且更有效，能够创建一个具有丰富信息和更好泛化能力的合成数据集。

基于不同主干网络和统计匹配的广义大规模数据压缩

Generalized Large-Scale Data Condensation via Various Backbone and  Statistical Matching

The ultimate goal of Dataset Distillation is to synthesize a small synthetic
dataset such that a model trained on this synthetic set will perform equally
well as a model trained on the full, real dataset. Until now, no method of
Dataset Distillation has reached this completely lossless goal, in part due to
the fact that previous methods only remain effective when the total number of
synthetic samples is extremely small. Since only so much information can be
contained in such a small number of samples, it seems that to achieve truly
loss dataset distillation, we must develop a distillation method that remains
effective as the size of the synthetic dataset grows. In this work, we present
such an algorithm and elucidate why existing methods fail to generate larger,
high-quality synthetic sets. Current state-of-the-art methods rely on
trajectory-matching, or optimizing the synthetic data to induce similar
long-term training dynamics as the real data. We empirically find that the
training stage of the trajectories we choose to match (i.e., early or late)
greatly affects the effectiveness of the distilled dataset. Specifically, early
trajectories (where the teacher network learns easy patterns) work well for a
low-cardinality synthetic set since there are fewer examples wherein to
distribute the necessary information. Conversely, late trajectories (where the
teacher network learns hard patterns) provide better signals for larger
synthetic sets since there are now enough samples to represent the necessary
complex patterns. Based on our findings, we propose to align the difficulty of
the generated patterns with the size of the synthetic dataset. In doing so, we
successfully scale trajectory matching-based methods to larger synthetic
datasets, achieving lossless dataset distillation for the very first time. Code
and distilled datasets are available at this https URL

开发一种能随着合成数据集规模增长而保持有效的新型数据集精馏方法，通过早期或晚期的轨迹匹配，成功将轨迹匹配方法扩展到更大的合成数据集，首次实现了无损数据集精馏。

基于难度对齐轨迹匹配的无损数据集蒸馏

Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory  Matching

Quantization is a promising approach for reducing the inference time and
memory footprint of neural networks. However, most existing quantization
methods require access to the original training dataset for retraining during
quantization. This is often not possible for applications with sensitive or
proprietary data, e.g., due to privacy and security concerns. Existing
zero-shot quantization methods use different heuristics to address this, but
they result in poor performance, especially when quantizing to ultra-low
precision. Here, we propose ZeroQ , a novel zero-shot quantization framework to
address this. ZeroQ enables mixed-precision quantization without any access to
the training or validation data. This is achieved by optimizing for a Distilled
Dataset, which is engineered to match the statistics of batch normalization
across different layers of the network. ZeroQ supports both uniform and
mixed-precision quantization. For the latter, we introduce a novel Pareto
frontier based method to automatically determine the mixed-precision bit
setting for all layers, with no manual search involved. We extensively test our
proposed method on a diverse set of models, including ResNet18/50/152,
MobileNetV2, ShuffleNet, SqueezeNext, and InceptionV3 on ImageNet, as well as
RetinaNet-ResNet50 on the Microsoft COCO dataset. In particular, we show that
ZeroQ can achieve 1.71\% higher accuracy on MobileNetV2, as compared to the
recently proposed DFQ method. Importantly, ZeroQ has a very low computational
overhead, and it can finish the entire quantization process in less than 30s
(0.5\% of one epoch training time of ResNet50 on ImageNet). We have
open-sourced the ZeroQ
framework\footnote{this https URL}.

本文提出了一种名为 ZeroQ 的新型零 - shot 量化框架，用于量化神经网络而无需访问原始数据，ZeroQ 通过优化一个与网络不同层的批量归一化统计数据相匹配的 Distilled Dataset 来实现。我们在不同的模型上进行了广泛测试，证明 ZeroQ 可以实现比 DFQ 方法更高的精度，具有极低的计算开销。