Data-free meta-learning (DFML) aims to enable efficient learning of new tasks
by meta-learning from a collection of pre-trained models without access to the
training data. Existing DFML work can only meta-learn from (i) white-box and
(ii) small-scale pre-trained models (iii) with the same architecture,
neglecting the more practical setting where the users only have inference
access to the APIs with arbitrary model architectures and model scale inside.
To solve this issue, we propose a Bi-level Data-free Meta Knowledge
Distillation (BiDf-MKD) framework to transfer more general meta knowledge from
a collection of black-box APIs to one single meta model. Specifically, by just
querying APIs, we inverse each API to recover its training data via a
zero-order gradient estimator and then perform meta-learning via a novel
bi-level meta knowledge distillation structure, in which we design a boundary
query set recovery technique to recover a more informative query set near the
decision boundary. In addition, to encourage better generalization within the
setting of limited API budgets, we propose task memory replay to diversify the
underlying task distribution by covering more interpolated tasks. Extensive
experiments in various real-world scenarios show the superior performance of
our BiDf-MKD framework.

提出一种 Bi-level Data-free Meta Knowledge Distillation (BiDf-MKD) 框架，通过 black-box APIs 进行元学习，通过边缘查询集回归技术恢复更详细的查询集，并设计任务记忆重放技术以在有限的 API 预算内促进更好的泛化。