Open-vocabulary semantic segmentation (OVS) aims to segment images of
arbitrary categories specified by class labels or captions. However, most
previous best-performing methods, whether pixel grouping methods or region
recognition methods, suffer from false matches between image features and
category labels. We attribute this to the natural gap between the textual
features and visual features. In this work, we rethink how to mitigate false
matches from the perspective of image-to-image matching and propose a novel
relation-aware intra-modal matching (RIM) framework for OVS based on visual
foundation models. RIM achieves robust region classification by firstly
constructing diverse image-modal reference features and then matching them with
region features based on relation-aware ranking distribution. The proposed RIM
enjoys several merits. First, the intra-modal reference features are better
aligned, circumventing potential ambiguities that may arise in cross-modal
matching. Second, the ranking-based matching process harnesses the structure
information implicit in the inter-class relationships, making it more robust
than comparing individually. Extensive experiments on three benchmarks
demonstrate that RIM outperforms previous state-of-the-art methods by large
margins, obtaining a lead of more than 10% in mIoU on PASCAL VOC benchmark.

基于视觉基于模型，提出了一种关系感知的单模态匹配 (OVS) 框架，用于解决像素分组和区域识别方法中图像特征与类别标签之间的误匹配问题，并在三个基准测试中展现出大幅度优势。

基于基础模型的图像匹配：开放词汇语义分割的新视角

Image-to-Image Matching via Foundation Models: A New Perspective for  Open-Vocabulary Semantic Segmentation

When exploring the development of Artificial General Intelligence (AGI), a
critical task for these models involves interpreting and processing information
from multiple image inputs. However, Large Multimodal Models (LMMs) encounter
two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a
tendency to blend information across multiple images. We first extensively
investigate the capability of LMMs to perceive fine-grained visual details when
dealing with multiple input images. The research focuses on two aspects: first,
image-to-image matching (to evaluate whether LMMs can effectively reason and
pair relevant images), and second, multi-image-to-text matching (to assess
whether LMMs can accurately capture and summarize detailed image information).
We conduct evaluations on a range of both open-source and closed-source large
models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model
performance, we further develop a Contrastive Chain-of-Thought (CoCoT)
prompting approach based on multi-input multimodal models. This method requires
LMMs to compare the similarities and differences among multiple image inputs,
and then guide the models to answer detailed questions about multi-image inputs
based on the identified similarities and differences. Our experimental results
showcase CoCoT's proficiency in enhancing the multi-image comprehension
capabilities of large multimodal models.

当探索人工通用智能（AGI）的发展时，大型多模态模型（LMMs）在处理多个图像输入的信息时面临两个问题：细粒度感知的缺乏和融合多个图像的倾向。我们首先广泛研究了 LMMs 在处理多个输入图像时感知细粒度视觉细节的能力。研究集中在两个方面：首先，图像与图像匹配（评估 LMMs 是否能够有效推理和配对相关图像），其次，多图像与文本匹配（评估 LMMs 是否能够准确捕捉和总结详细的图像信息）。我们对一系列开源和闭源的大型模型进行评估，包括 GPT-4V，Gemini，OpenFlamingo 和 MMICL。为了增强模型性能，我们还基于多输入多模态模型开发了一种对比的思维链（CoCoT）启发方法。该方法要求 LMMs 比较多个图像输入的相似性和差异性，然后根据确定的相似性和差异性指导模型回答关于多图像输入的详细问题。我们的实验结果展示了 CoCoT 在增强大型多模态模型的多图像理解能力方面的熟练度。