Even when using large multi-modal foundation models, few-shot learning is
still challenging -- if there is no proper inductive bias, it is nearly
impossible to keep the nuanced class attributes while removing the visually
prominent attributes that spuriously correlate with class labels. To this end,
we find an inductive bias that the time-steps of a Diffusion Model (DM) can
isolate the nuanced class attributes, i.e., as the forward diffusion adds noise
to an image at each time-step, nuanced attributes are usually lost at an
earlier time-step than the spurious attributes that are visually prominent.
Building on this, we propose Time-step Few-shot (TiF) learner. We train
class-specific low-rank adapters for a text-conditioned DM to make up for the
lost attributes, such that images can be accurately reconstructed from their
noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt
are essentially a parameterization of only the nuanced class attributes. For a
test image, we can use the parameterization to only extract the nuanced class
attributes for classification. TiF learner significantly outperforms OpenCLIP
and its adapters on a variety of fine-grained and customized few-shot learning
tasks. Codes are in this https URL

通过使用时间步骤能够隔离微妙的类属性，我们提出了一种用于少样本学习的时间步骤学习器，该学习器通过训练特定于每个类别的低秩适配器来补偿丢失的属性，从而在各种细粒度和定制的少样本学习任务上显著优于 OpenCLIP 及其适配器。

少样本学习者参数化的扩散时间步

Few-shot Learner Parameterization by Diffusion Time-steps

Modern classification problems exhibit heterogeneities across individual
classes: Each class may have unique attributes, such as sample size, label
quality, or predictability (easy vs difficult), and variable importance at
test-time. Without care, these heterogeneities impede the learning process,
most notably, when optimizing fairness objectives. Confirming this, under a
gaussian mixture setting, we show that the optimal SVM classifier for balanced
accuracy needs to be adaptive to the class attributes. This motivates us to
propose CAP: An effective and general method that generates a class-specific
learning strategy (e.g. hyperparameter) based on the attributes of that class.
This way, optimization process better adapts to heterogeneities. CAP leads to
substantial improvements over the naive approach of assigning separate
hyperparameters to each class. We instantiate CAP for loss function design and
post-hoc logit adjustment, with emphasis on label-imbalanced problems. We show
that CAP is competitive with prior art and its flexibility unlocks clear
benefits for fairness objectives beyond balanced accuracy. Finally, we evaluate
CAP on problems with label noise as well as weighted test objectives to
showcase how CAP can jointly adapt to different heterogeneities.

现代分类问题存在类内异质性：每个类别可能具有唯一的属性，如样本大小、标签质量或可预测性（易 vs 难），以及测试时的变量重要性。我们提出了 CAP 方法，该方法根据类别属性生成特定于类别的学习策略（例如超参数），优化过程更好地适应这些异质性。CAP 在损失函数设计和事后 logit 调整方面具有竞争力，并在公平目标超过平衡准确性时展现明显优势。最后，我们评估了 CAP 方法在存在标签噪声和加权测试目标的问题上，展示了 CAP 方法如何同时适应不同的异质性。

类属性先验：将优化适应异质性和公平目标

Class-attribute Priors: Adapting Optimization to Heterogeneity and  Fairness Objective

Zero-shot learning (ZSL) aims to classify images of an unseen class only
based on a few attributes describing that class but no access to any training
sample. A popular strategy is to learn a mapping between the semantic space of
class attributes and the visual space of images based on the seen classes and
their data. Thus, an unseen class image can be ideally mapped to its
corresponding class attributes. The key challenge is how to align the
representations in the two spaces. For most ZSL settings, the attributes for
each seen/unseen class are only represented by a vector while the seen-class
data provide much more information. Thus, the imbalanced supervision from the
semantic and the visual space can make the learned mapping easily overfitting
to the seen classes. To resolve this problem, we propose Isometric Propagation
Network (IPN), which learns to strengthen the relation between classes within
each space and align the class dependency in the two spaces. Specifically, IPN
learns to propagate the class representations on an auto-generated graph within
each space. In contrast to only aligning the resulted static representation, we
regularize the two dynamic propagation procedures to be isometric in terms of
the two graphs' edge weights per step by minimizing a consistency loss between
them. IPN achieves state-of-the-art performance on three popular ZSL
benchmarks. To evaluate the generalization capability of IPN, we further build
two larger benchmarks with more diverse unseen classes and demonstrate the
advantages of IPN on them.

该研究提出了 Isometric Propagation Network（IPN），通过学习在语义空间和视觉空间内每个类别的表示之间的联系来解决零样本学习中的显着问题，并在三个流行的零样本学习基准测试中取得了最先进的表现。