Low-shot image classification is a fundamental task in computer vision, and
the emergence of large-scale vision-language models such as CLIP has greatly
advanced the forefront of research in this field. However, most existing
CLIP-based methods lack the flexibility to effectively incorporate other
pre-trained models that encompass knowledge distinct from CLIP. To bridge the
gap, this work proposes a simple and effective probabilistic model ensemble
framework based on Gaussian processes, which have previously demonstrated
remarkable efficacy in processing small data. We achieve the integration of
prior knowledge by specifying the mean function with CLIP and the kernel
function with an ensemble of deep kernels built upon various pre-trained
models. By regressing the classification label directly, our framework enables
analytical inference, straightforward uncertainty quantification, and
principled hyper-parameter tuning. Through extensive experiments on standard
benchmarks, we demonstrate that our method consistently outperforms competitive
ensemble baselines regarding predictive performance. Additionally, we assess
the robustness of our method and the quality of the yielded uncertainty
estimates on out-of-distribution datasets. We also illustrate that our method,
despite relying on label regression, still enjoys superior model calibration
compared to most deterministic baselines.

基于高斯过程的概率模型集成框架，通过结合 CLIP 和基于各种预训练模型构建的深度核函数集成，实现了先验知识的整合，并在标准基准上通过广泛实验证明了我们方法在预测性能方面始终优于竞争方法。

基于贝叶斯的预训练模型在低 - shot 图像分类中的探索

Bayesian Exploration of Pre-trained Models for Low-shot Image  Classification

Low-shot image classification, where training images are limited or
inaccessible, has benefited from recent progress on pre-trained vision-language
(VL) models with strong generalizability, e.g. CLIP. Prompt learning methods
built with VL models generate text features from the class names that only have
confined class-specific information. Large Language Models (LLMs), with their
vast encyclopedic knowledge, emerge as the complement. Thus, in this paper, we
discuss the integration of LLMs to enhance pre-trained VL models, specifically
on low-shot classification. However, the domain gap between language and vision
blocks the direct application of LLMs. Thus, we propose LLaMP, Large Language
Models as Prompt learners, that produces adaptive prompts for the CLIP text
encoder, establishing it as the connecting bridge. Experiments show that,
compared with other state-of-the-art prompt learning methods, LLaMP yields
better performance on both zero-shot generalization and few-shot image
classification, over a spectrum of 11 datasets.

通过整合大型语言模型（LLMs）提升预训练视觉 - 语言模型（VL）在低样本图像分类中的能力，提出了大型语言模型作为提示学习者（LLaMP）的方法，并在 11 个数据集上的零样本和小样本图像分类任务中取得了更好的性能。

大型语言模型是低样本图像分类的良好触发器学习者

Large Language Models are Good Prompt Learners for Low-Shot Image  Classification

We propose Masked Siamese Networks (MSN), a self-supervised learning
framework for learning image representations. Our approach matches the
representation of an image view containing randomly masked patches to the
representation of the original unmasked image. This self-supervised
pre-training strategy is particularly scalable when applied to Vision
Transformers since only the unmasked patches are processed by the network. As a
result, MSNs improve the scalability of joint-embedding architectures, while
producing representations of a high semantic level that perform competitively
on low-shot image classification. For instance, on ImageNet-1K, with only 5,000
annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1%
of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new
state-of-the-art for self-supervised learning on this benchmark. Our code is
publicly available.

提出了一种自监督学习框架 ——Masked Siamese Networks (MSN)，用于学习图像表示。通过将随机遮罩块的图像视图的表征与原始未遮罩图像的表征进行匹配，可以进行自监督预训练，该策略特别适用于 Vision Transformer 模型，在提高联合嵌入结构的可扩展性的同时，产生高语义级别的表示，并在低样本图像分类方面表现竞争性，对 ImageNet-1K 数据集上，使用基础 MSN 模型以仅有 5000 个带注释的图像，达到了 72.4％的 top-1 准确度，并且只有 1％的 ImageNet-1K 标签，就取得了 75.7％的 top-1 准确度，创立了该基准测试的自监督学习的新的最佳成果。