Despite the success of two-stage few-shot classification methods, in the
episodic meta-training stage, the model suffers severe overfitting. We
hypothesize that it is caused by over-discrimination, i.e., the model learns to
over-rely on the superficial features that fit for base class discrimination
while suppressing the novel class generalization. To penalize
over-discrimination, we introduce knowledge distillation techniques to keep
novel generalization knowledge from the teacher model during training.
Specifically, we select the teacher model as the one with the best validation
accuracy during meta-training and restrict the symmetric Kullback-Leibler (SKL)
divergence between the output distribution of the linear classifier of the
teacher model and that of the student model. This simple approach outperforms
the standard meta-training process. We further propose the Nearest Neighbor
Symmetric Kullback-Leibler (NNSKL) divergence for meta-training to push the
limits of knowledge distillation techniques. NNSKL takes few-shot tasks as
input and penalizes the output of the nearest neighbor classifier, which
possesses an impact on the relationships between query embedding and support
centers. By combining SKL and NNSKL in meta-training, the model achieves even
better performance and surpasses state-of-the-art results on several
benchmarks.

介绍了一种通过知识蒸馏减少过度区分的元学习方法，并结合最近邻对称 KL 散度进行训练，在几个基准测试中取得了最先进的结果。

理解情节元训练的过拟合

Understanding the Overfitting of the Episodic Meta-training

A new form of the variational autoencoder (VAE) is proposed, based on the
symmetric Kullback-Leibler divergence. It is demonstrated that learning of the
resulting symmetric VAE (sVAE) has close connections to previously developed
adversarial-learning methods. This relationship helps unify the previously
distinct techniques of VAE and adversarially learning, and provides insights
that allow us to ameliorate shortcomings with some previously developed
adversarial methods. In addition to an analysis that motivates and explains the
sVAE, an extensive set of experiments validate the utility of the approach.

提出了一种新形式的变分自动编码器 (VAE)，其基于对称的 Kullback-Leibler 散度，证明了学习其结果的对称 VAE (sVAE) 与先前开发的对抗性学习方法有密切的联系，对 VAE 和对抗性学习之前的不同技术进行统一，提供了见解，自此不需要先前一些对抗性方法的缺陷。除了激励和解释 sVAE 的分析外，还有一系列广泛的实验验证了该方法的效用。