The task of few-shot image classification and segmentation (FS-CS) requires the classification and segmentation of target objects in a query image, given only a few examples of the target classes. We introduce a method that utilises large language models (LLM) as an agent to address the FS-CS problem in a training-free manner. By making the LLM the task planner and off-the-shelf vision models the tools, the proposed method is capable of classifying and segmenting target objects using only image-level labels. Specifically, chain-of-thought prompting and in-context learning guide the LLM to observe support images like human; vision models such as Segment Anything Model (SAM) and GPT-4Vision assist LLM understand spatial and semantic information at the same time. Ultimately, the LLM uses its summarizing and reasoning capabilities to classify and segment the query image. The proposed method's modular framework makes it easily extendable. Our approach achieves state-of-the-art performance on the Pascal-5i dataset.

利用大型语言模型作为任务规划者，视觉模型作为工具，通过观察支持图像和使用语境学习指导大型语言模型，利用其总结和推理能力对查询图像进行分类和分割，从而在无需训练的情况下解决了少样本图像分类和分割问题，并在Pascal-5i数据集上取得了最先进的性能。

基于大语言模型的小样本分类与分割代理