We present Self-Context Adaptation (SeCAt), a self-supervised approach that
unlocks open-ended few-shot abilities of small visual language models. Our
proposed adaptation algorithm explicitly learns from symbolic, yet
self-supervised training tasks. Specifically, our approach imitates image
captions in a self-supervised way based on clustering a large pool of images
followed by assigning semantically-unrelated names to clusters. By doing so, we
construct the `self-context', a training signal consisting of interleaved
sequences of image and pseudo-caption pairs and a query image for which the
model is trained to produce the right pseudo-caption. We demonstrate the
performance and flexibility of SeCAt on several multimodal few-shot datasets,
spanning various granularities. By using models with approximately 1B
parameters we outperform the few-shot abilities of much larger models, such as
Frozen and FROMAGe. SeCAt opens new possibilities for research in open-ended
few-shot learning that otherwise requires access to large or proprietary
models.

Self-Context Adaptation (SeCAt) 是一种自供给的方法，通过学习符号化的、自供给的训练任务，解锁了小型视觉语言模型的开放性少样本能力。我们的方法模仿图像的自供应方式，通过对大量图像进行聚类，为聚类分配语义不相关的名称，构建了 “自上下文”，即由图像和伪标题对序列交替和查询图像组成的训练信号，模型要训练出正确的伪标题。我们在几个多模态少样本数据集上展示了 SeCAt 的性能和灵活性，涵盖了各种粒度。通过使用大约 10 亿个参数的模型，我们的性能优于更大的模型，如 Frozen 和 FROMAGe，为开放性少样本学习中需要访问大型或专有模型的研究开辟了新的可能性。