Human learning benefits from multi-modal inputs that often appear as rich
semantics (e.g., description of an object's attributes while learning about
it). This enables us to learn generalizable concepts from very limited visual
examples. However, current few-shot learning (FSL) methods use numerical class
labels to denote object classes which do not provide rich semantic meanings
about the learned concepts. In this work, we show that by using 'class-level'
language descriptions, that can be acquired with minimal annotation cost, we
can improve the FSL performance. Given a support set and queries, our main idea
is to create a bottleneck visual feature (hybrid prototype) which is then used
to generate language descriptions of the classes as an auxiliary task during
training. We develop a Transformer based forward and backward encoding
mechanism to relate visual and semantic tokens that can encode intricate
relationships between the two modalities. Forcing the prototypes to retain
semantic information about class description acts as a regularizer on the
visual features, improving their generalization to novel classes at inference.
Furthermore, this strategy imposes a human prior on the learned
representations, ensuring that the model is faithfully relating visual and
semantic concepts, thereby improving model interpretability. Our experiments on
four datasets and ablation studies show the benefit of effectively modeling
rich semantics for FSL.

利用少量的『类别级别』的语言描述，结合视觉特征分形成一个瓶颈视觉特征 (混合原型) 并建立一种 Transformer 机制，以编码这两种形式的丰富语义，并且经过多个数据集的实验证明，该算法能有效提升 few-shot learning 的性能。