Existing data-free model stealing methods use a generator to produce samples
in order to train a student model to match the target model outputs. To this
end, the two main challenges are estimating gradients of the target model
without access to its parameters, and generating a diverse set of training
samples that thoroughly explores the input space. We propose a Dual Student
method where two students are symmetrically trained in order to provide the
generator a criterion to generate samples that the two students disagree on. On
one hand, disagreement on a sample implies at least one student has classified
the sample incorrectly when compared to the target model. This incentive
towards disagreement implicitly encourages the generator to explore more
diverse regions of the input space. On the other hand, our method utilizes
gradients of student models to indirectly estimate gradients of the target
model. We show that this novel training objective for the generator network is
equivalent to optimizing a lower bound on the generator's loss if we had access
to the target model gradients. We show that our new optimization framework
provides more accurate gradient estimation of the target model and better
accuracies on benchmark classification datasets. Additionally, our approach
balances improved query efficiency with training computation cost. Finally, we
demonstrate that our method serves as a better proxy model for transfer-based
adversarial attacks than existing data-free model stealing methods.

通过对称地训练两个学生模型，提供生成器一种生成样本的标准，使两个学生模型对样本有分歧，这个方法可以间接地估计目标模型的梯度，优化生成器网络的训练目标，并提供更准确的目标模型梯度估计和更好的基准分类数据集准确性。此外，在改进查询效率的同时，我们的方法仍具有训练计算成本的平衡，有效用于转移性对抗攻击的代理模型。