Large vision-language models revolutionized image classification and semantic
segmentation paradigms. However, they typically assume a pre-defined set of
categories, or vocabulary, at test time for composing textual prompts. This
assumption is impractical in scenarios with unknown or evolving semantic
context. Here, we address this issue and introduce the Vocabulary-free Image
Classification (VIC) task, which aims to assign a class from an unconstrained
language-induced semantic space to an input image without needing a known
vocabulary. VIC is challenging due to the vastness of the semantic space, which
contains millions of concepts, including fine-grained categories. To address
VIC, we propose Category Search from External Databases (CaSED), a
training-free method that leverages a pre-trained vision-language model and an
external database. CaSED first extracts the set of candidate categories from
the most semantically similar captions in the database and then assigns the
image to the best-matching candidate category according to the same
vision-language model. Furthermore, we demonstrate that CaSED can be applied
locally to generate a coarse segmentation mask that classifies image regions,
introducing the task of Vocabulary-free Semantic Segmentation. CaSED and its
variants outperform other more complex vision-language models, on
classification and semantic segmentation benchmarks, while using much fewer
parameters.

通过引入外部数据库，提出了一种无需预定义类别词汇即可对图像进行分类和语义分割的方法，该方法基于预训练的视觉 - 语言模型，使用较少参数的情况下在分类和语义分割基准上优于其他更复杂的模型。

无词汇图像分类与语义分割

Vocabulary-free Image Classification and Semantic Segmentation

Recent advances in large vision-language models have revolutionized the image
classification paradigm. Despite showing impressive zero-shot capabilities, a
pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time
for composing the textual prompts. However, such assumption can be impractical
when the semantic context is unknown and evolving. We thus formalize a novel
task, termed as Vocabulary-free Image Classification (VIC), where we aim to
assign to an input image a class that resides in an unconstrained
language-induced semantic space, without the prerequisite of a known
vocabulary. VIC is a challenging task as the semantic space is extremely large,
containing millions of concepts, with hard-to-discriminate fine-grained
categories. In this work, we first empirically verify that representing this
semantic space by means of an external vision-language database is the most
effective way to obtain semantically relevant content for classifying the
image. We then propose Category Search from External Databases (CaSED), a
method that exploits a pre-trained vision-language model and an external
vision-language database to address VIC in a training-free manner. CaSED first
extracts a set of candidate categories from captions retrieved from the
database based on their semantic similarity to the image, and then assigns to
the image the best matching candidate category according to the same
vision-language model. Experiments on benchmark datasets validate that CaSED
outperforms other complex vision-language frameworks, while being efficient
with much fewer parameters, paving the way for future research in this
direction.

该论文提出了一种名为 “无词汇图像分类（VIC）” 的新任务，通过使用外部视觉语言数据库（即语义空间）来解决这个任务，为此提出了一个名为 CaSED 的方法，它利用预训练的视觉语言模型和外部视觉语言数据库在训练自由的情况下来完成任务，与其他复杂的视觉语言框架相比，在参数更少的情况下具有更好的分类效果。