The task of few-shot image classification and segmentation (FS-CS) involves
classifying and segmenting target objects in a query image, given only a few
examples of the target classes. We introduce the Vision-Instructed Segmentation
and Evaluation (VISE) method that transforms the FS-CS problem into the Visual
Question Answering (VQA) problem, utilising Vision-Language Models (VLMs), and
addresses it in a training-free manner. By enabling a VLM to interact with
off-the-shelf vision models as tools, the proposed method is capable of
classifying and segmenting target objects using only image-level labels.
Specifically, chain-of-thought prompting and in-context learning guide the VLM
to answer multiple-choice questions like a human; vision models such as YOLO
and Segment Anything Model (SAM) assist the VLM in completing the task. The
modular framework of the proposed method makes it easily extendable. Our
approach achieves state-of-the-art performance on the Pascal-5i and COCO-20i
datasets.

用视觉引导的分割和评估方法将少样本图像分类和分割问题转化为视觉问答问题，利用视觉语言模型以无需训练的方式进行解决，并通过交互式学习和模块化框架达到最先进的性能。