The task of few-shot image classification and segmentation (FS-CS) requires
the classification and segmentation of target objects in a query image, given
only a few examples of the target classes. We introduce a method that utilises
large language models (LLM) as an agent to address the FS-CS problem in a
training-free manner. By making the LLM the task planner and off-the-shelf
vision models the tools, the proposed method is capable of classifying and
segmenting target objects using only image-level labels. Specifically,
chain-of-thought prompting and in-context learning guide the LLM to observe
support images like human; vision models such as Segment Anything Model (SAM)
and GPT-4Vision assist LLM understand spatial and semantic information at the
same time. Ultimately, the LLM uses its summarizing and reasoning capabilities
to classify and segment the query image. The proposed method's modular
framework makes it easily extendable. Our approach achieves state-of-the-art
performance on the Pascal-5i dataset.

利用大型语言模型作为任务规划者，视觉模型作为工具，通过观察支持图像和使用语境学习指导大型语言模型，利用其总结和推理能力对查询图像进行分类和分割，从而在无需训练的情况下解决了少样本图像分类和分割问题，并在 Pascal-5i 数据集上取得了最先进的性能。

基于大语言模型的小样本分类与分割代理

Few-Shot Classification & Segmentation Using Large Language Models Agent

We introduce a new setting of Novel Class Discovery in Semantic Segmentation
(NCDSS), which aims at segmenting unlabeled images containing new classes given
prior knowledge from a labeled set of disjoint classes. In contrast to existing
approaches that look at novel class discovery in image classification, we focus
on the more challenging semantic segmentation. In NCDSS, we need to distinguish
the objects and background, and to handle the existence of multiple classes
within an image, which increases the difficulty in using the unlabeled data. To
tackle this new setting, we leverage the labeled base data and a saliency model
to coarsely cluster novel classes for model training in our basic framework.
Additionally, we propose the Entropy-based Uncertainty Modeling and
Self-training (EUMS) framework to overcome noisy pseudo-labels, further
improving the model performance on the novel classes. Our EUMS utilizes an
entropy ranking technique and a dynamic reassignment to distill clean labels,
thereby making full use of the noisy data via self-supervised learning. We
build the NCDSS benchmark on the PASCAL-5$^i$ dataset and COCO-20$^i$ dataset.
Extensive experiments demonstrate the feasibility of the basic framework
(achieving an average mIoU of 49.81% on PASCAL-5$^i$) and the effectiveness of
EUMS framework (outperforming the basic framework by 9.28% mIoU on
PASCAL-5$^i$).

这篇论文介绍了一种新的新类发现语义分割方法，使用基于熵的不确定性建模和自训练技术来提高模型性能，实验结果展示了这种方法对于 PASCAL-5i 数据集上的平均 IoU 提高了 9.28%。

语义分割中的新类别发现

Novel Class Discovery in Semantic Segmentation

Few-shot segmentation (FSS) methods perform image segmentation for a
particular object class in a target (query) image, using a small set of
(support) image-mask pairs. Recent deep neural network based FSS methods
leverage high-dimensional feature similarity between the foreground features of
the support images and the query image features. In this work, we demonstrate
gaps in the utilization of this similarity information in existing methods, and
present a framework - SimPropNet, to bridge those gaps. We propose to jointly
predict the support and query masks to force the support features to share
characteristics with the query features. We also propose to utilize
similarities in the background regions of the query and support images using a
novel foreground-background attentive fusion mechanism. Our method achieves
state-of-the-art results for one-shot and five-shot segmentation on the
PASCAL-5i dataset. The paper includes detailed analysis and ablation studies
for the proposed improvements and quantitative comparisons with contemporary
methods.

本文提出了一种叫做 SimPropNet 的框架，利用前景背景关注融合机制，同时预测支持和查询掩模，以共享特征信息来填补现有方法在利用相似性信息方面的不足，其在 PASCAL-5i 数据集上实现了最先进的一次和五次分割结果。