Few-shot image classification involves classifying images using very few
training examples. Recent vision foundation models show excellent few-shot
transfer abilities, but are large and slow at inference. Using knowledge
distillation, the capabilities of high-performing but slow models can be
transferred to tiny, efficient models. However, common distillation methods
require a large set of unlabeled data, which is not available in the few-shot
setting. To overcome this lack of data, there has been a recent interest in
using synthetic data.
We expand on this work by presenting a novel diffusion model inversion
technique (TINT) combining the diversity of textual inversion with the
specificity of null-text inversion. Using this method in a few-shot
distillation pipeline leads to state-of-the-art accuracy among small student
models on popular benchmarks, while being significantly faster than prior work.
This allows us to push even tiny models to high accuracy using only a tiny
application-specific dataset, albeit relying on extra data for pre-training.
Popular few-shot benchmarks involve evaluation over a large number of
episodes, which is computationally cumbersome for methods involving synthetic
data generation. Therefore, we also present a theoretical analysis on how the
variance of the accuracy estimator depends on the number of episodes and query
examples, and use these results to lower the computational effort required for
method evaluation. In addition, to further motivate the use of generative
models in few-shot distillation, we demonstrate that our method performs better
compared to training on real data mined from the dataset used to train the
diffusion model.
Source code will be made available at this https URL

使用少量训练样本进行图像分类的研究涉及到利用知识蒸馏将高性能但计算速度较慢的模型的能力转移到小型高效模型上，通过使用合成数据来弥补数据不足，并利用扩散模型反演技术（TINT）和生成模型进行少样本蒸馏，达到接近先前研究的最高准确性，同时比先前研究更快速地完成任务。