Dataset distillation methods have demonstrated remarkable performance for
neural networks trained with very limited training data. However, a significant
challenge arises in the form of architecture overfitting: the distilled
training data synthesized by a specific network architecture (i.e., training
network) generates poor performance when trained by other network architectures
(i.e., test networks). This paper addresses this issue and proposes a series of
approaches in both architecture designs and training schemes which can be
adopted together to boost the generalization performance across different
network architectures on the distilled training data. We conduct extensive
experiments to demonstrate the effectiveness and generality of our methods.
Particularly, across various scenarios involving different sizes of distilled
data, our approaches achieve comparable or superior performance to existing
methods when training on the distilled data using networks with larger
capacities.

通过提出一系列的方法，本文解决了基于特定网络结构训练的合成训练数据在其他网络结构中训练时性能变差的问题，从而提高了在不同网络结构上通过合成训练数据的泛化性能。通过广泛的实验证明了我们的方法的有效性和通用性。