Large Pre-trained Transformers exhibit an intriguing capacity for in-context
learning. Without gradient updates, these models can rapidly construct new
predictors from demonstrations presented in the inputs. Recent works promote
this ability in the vision-language domain by incorporating visual information
into large language models that can already make in-context predictions.
However, these methods could inherit issues in the language domain, such as
template sensitivity and hallucination. Also, the scale of these language
models raises a significant demand for computations, making learning and
operating these models resource-intensive. To this end, we raise a question:
``How can we enable in-context learning for general models without being
constrained on large language models?". To answer it, we propose a succinct and
general framework, Self-supervised IN-Context learning (SINC), that introduces
a meta-model to learn on self-supervised prompts consisting of tailored
demonstrations. The learned models can be transferred to downstream tasks for
making in-context predictions on-the-fly. Extensive experiments show that SINC
outperforms gradient-based methods in various vision-language tasks under
few-shot settings. Furthermore, the designs of SINC help us investigate the
benefits of in-context learning across different tasks, and the analysis
further reveals the essential components for the emergence of in-context
learning in the vision-language domain.

本文提出了一种自监督的上下文学习 (SINC) 框架，可以在视觉 - 语言领域的各种任务中进行前馈预测，无需通过渐变更新进行特定任务微调，并表明在少样本数据的情况下 SINC 方法优于基于梯度的方法。