Prompt ensembling of Large Language Model (LLM) generated category-specific
prompts has emerged as an effective method to enhance zero-shot recognition
ability of Vision-Language Models (VLMs). To obtain these category-specific
prompts, the present methods rely on hand-crafting the prompts to the LLMs for
generating VLM prompts for the downstream tasks. However, this requires
manually composing these task-specific prompts and still, they might not cover
the diverse set of visual concepts and task-specific styles associated with the
categories of interest. To effectively take humans out of the loop and
completely automate the prompt generation process for zero-shot recognition, we
propose Meta-Prompting for Visual Recognition (MPVR). Taking as input only
minimal information about the target task, in the form of its short natural
language description, and a list of associated class labels, MPVR automatically
produces a diverse set of category-specific prompts resulting in a strong
zero-shot classifier. MPVR generalizes effectively across various popular
zero-shot image recognition benchmarks belonging to widely different domains
when tested with multiple LLMs and VLMs. For example, MPVR obtains a zero-shot
recognition improvement over CLIP by up to 19.8% and 18.2% (5.0% and 4.5% on
average over 20 datasets) leveraging GPT and Mixtral LLMs, respectively

通过 Meta-Prompting for Visual Recognition (MPVR) 方法，仅凭目标任务的简短自然语言描述和相关类别标签的最小信息输入，自动产生一组多样的类别特定提示，从而实现强大的零样本分类器。在多个不同领域的流行的零样本图像识别基准上，使用多个 LLMs 和 VLMs 测试，MPVR 可以有效地推广，比 CLIP 提高了最高 19.8％和 18.2％（平均 20 个数据集上分别为 5.0％和 4.5％），依赖于 GPT 和 Mixtral LLMs。

利用 LLMs 自动化零射视觉识别的元提示

Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs

Foundational vision-language models such as CLIP are becoming a new paradigm
in vision, due to their excellent generalization abilities. However, adapting
these models for downstream tasks while maintaining their generalization
remains a challenge. In literature, one branch of methods adapts CLIP by
learning prompts using visual information. While effective, most of these works
require labeled data which is not practical, and often struggle to generalize
towards new datasets due to over-fitting on the source data. An alternative
approach resorts to training-free methods by generating class descriptions from
large language models (LLMs) and perform prompt ensembling. However, these
methods often generate class specific prompts that cannot be transferred to
other classes, which incur higher costs by generating LLM descriptions for each
class separately. In this work, we propose to combine the strengths of these
both streams of methods by learning prompts using only text data derived from
LLMs. As supervised training of prompts is not trivial due to absence of
images, we develop a training approach that allows prompts to extract rich
contextual knowledge from LLM data. Moreover, with LLM contextual data mapped
within the learned prompts, it enables zero-shot transfer of prompts to new
classes and datasets potentially cutting the LLM prompt engineering cost. To
the best of our knowledge, this is the first work that learns generalized
prompts using text only data. We perform extensive evaluations on 4 benchmarks
where our method improves over prior ensembling works while being competitive
to those utilizing labeled images. Our code and pre-trained models are
available at this https URL

本研究提出了一种使用仅文本数据学习通用提示的方法，通过将 LLL（large language models）中的上下文数据映射到学习到的提示中，实现了零样本转移，并在 4 个基准测试上进行了广泛评估。

仅使用文本监督在视觉 - 语言模型中学习提示

Learning to Prompt with Text Only Supervision for Vision-Language Models

Methods such as chain-of-thought prompting and self-consistency have pushed
the frontier of language model reasoning performance with no additional
training. To further improve performance, we propose a prompt ensembling method
for large language models, which uses a small dataset to construct a set of few
shot prompts that together comprise a ``boosted prompt ensemble''. The few shot
examples for each prompt are chosen in a stepwise fashion to be ``hard''
examples on which the previous step's ensemble is uncertain. We show that this
outperforms single-prompt output-space ensembles and bagged prompt-space
ensembles on the GSM8k and AQuA datasets, among others. We propose both
train-time and test-time versions of boosted prompting that use different
levels of available annotation and conduct a detailed empirical study of our
algorithm.

我们提出了一种基于提示集成的方法来进一步提高语言模型推理性能，并在 GSM8k 和 AQuA 数据集上的实验中验证了该方法的优越性。