Dataset condensation can be used to reduce the computational cost of training
multiple models on a large dataset by condensing the training dataset into a
small synthetic set. State-of-the-art approaches rely on matching the model
gradients between the real and synthetic data. However, there is no theoretical
guarantee of the generalizability of the condensed data: data condensation
often generalizes poorly across hyperparameters/architectures in practice. This
paper considers a different condensation objective specifically geared toward
hyperparameter search. We aim to generate a synthetic validation dataset so
that the validation-performance rankings of the models, with different
hyperparameters, on the condensed and original datasets are comparable. We
propose a novel hyperparameter-calibrated dataset condensation (HCDC)
algorithm, which obtains the synthetic validation dataset by matching the
hyperparameter gradients computed via implicit differentiation and efficient
inverse Hessian approximation. Experiments demonstrate that the proposed
framework effectively maintains the validation-performance rankings of models
and speeds up hyperparameter/architecture search for tasks on both images and
graphs.

本文提出了一种针对超参数搜索的新型超参数校准数据集压缩（HCDC）算法，通过匹配通过隐式微分和高效逆 Hessian 逼近计算的超参数梯度来生成合成的验证数据集，实验证明该框架有效地维持了模型的验证性能排名，并加速了基于图像和图形的超参数 / 架构搜索任务。