Recent papers have suggested that transfer learning can outperform
sophisticated meta-learning methods for few-shot image classification. We take
this hypothesis to its logical conclusion, and suggest the use of an ensemble
of high-quality, pre-trained feature extractors for few-shot image
classification. We show experimentally that a library of pre-trained feature
extractors combined with a simple feed-forward network learned with an
L2-regularizer can be an excellent option for solving cross-domain few-shot
image classification. Our experimental results suggest that this simpler
sample-efficient approach far outperforms several well-established
meta-learning algorithms on a variety of few-shot tasks.

本研究探讨使用预训练特征提取器的集成方法来解决跨域少样本图像分类问题，实验结果表明这种简单且高效的方法在多个少样本任务上都能胜过复杂的元学习算法。

Few-shot 图像分类：使用预训练特征提取器库和简单分类器

Few-shot Image Classification: Just Use a Library of Pre-trained Feature  Extractors and a Simple Classifier

Pre-trained universal feature extractors, such as BERT for natural language
processing and VGG for computer vision, have become effective methods for
improving deep learning models without requiring more labeled data. While
effective, feature extractors like BERT may be prohibitively large for some
deployment scenarios. We explore weight pruning for BERT and ask: how does
compression during pre-training affect transfer learning? We find that pruning
affects transfer learning in three broad regimes. Low levels of pruning
(30-40%) do not affect pre-training loss or transfer to downstream tasks at
all. Medium levels of pruning increase the pre-training loss and prevent useful
pre-training information from being transferred to downstream tasks. High
levels of pruning additionally prevent models from fitting downstream datasets,
leading to further degradation. Finally, we observe that fine-tuning BERT on a
specific task does not improve its prunability. We conclude that BERT can be
pruned once during pre-training rather than separately for each task without
affecting performance.

本文探讨使用权重剪枝压缩 BERT 模型对迁移学习的影响，研究结果发现，对于预训练模型进行低水平的剪枝不会对其迁移能力造成影响，而高水平的剪枝则会导致模型性能的下降。在指定任务中对 BERT 进行微调并不会对其剪枝产生帮助。因此，本文得出结论：可以在预训练期间对 BERT 进行一次剪枝并保持其性能。