Even when using large multi-modal foundation models, few-shot learning is
still challenging -- if there is no proper inductive bias, it is nearly
impossible to keep the nuanced class attributes while removing the visually
prominent attributes that spuriously correlate with class labels. To this end,
we find an inductive bias that the time-steps of a Diffusion Model (DM) can
isolate the nuanced class attributes, i.e., as the forward diffusion adds noise
to an image at each time-step, nuanced attributes are usually lost at an
earlier time-step than the spurious attributes that are visually prominent.
Building on this, we propose Time-step Few-shot (TiF) learner. We train
class-specific low-rank adapters for a text-conditioned DM to make up for the
lost attributes, such that images can be accurately reconstructed from their
noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt
are essentially a parameterization of only the nuanced class attributes. For a
test image, we can use the parameterization to only extract the nuanced class
attributes for classification. TiF learner significantly outperforms OpenCLIP
and its adapters on a variety of fine-grained and customized few-shot learning
tasks. Codes are in this https URL

通过使用时间步骤能够隔离微妙的类属性，我们提出了一种用于少样本学习的时间步骤学习器，该学习器通过训练特定于每个类别的低秩适配器来补偿丢失的属性，从而在各种细粒度和定制的少样本学习任务上显著优于 OpenCLIP 及其适配器。