In this paper, we highlight a problem of evaluation metrics adopted in the
open-vocabulary segmentation. That is, the evaluation process still heavily
relies on closed-set metrics on zero-shot or cross-dataset pipelines without
considering the similarity between predicted and ground truth categories. To
tackle this issue, we first survey eleven similarity measurements between two
categorical words using WordNet linguistics statistics, text embedding, and
language models by comprehensive quantitative analysis and user study. Built
upon those explored measurements, we designed novel evaluation metrics, namely
Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary
segmentation tasks. We benchmarked the proposed evaluation metrics on 12
open-vocabulary methods of three segmentation tasks. Even though the relative
subjectivity of similarity distance, we demonstrate that our metrics can still
well evaluate the open ability of the existing open-vocabulary segmentation
methods. We hope that our work can bring with the community new thinking about
how to evaluate the open ability of models. The evaluation code is released in
github.

评估在开放词汇分割中采用的评估指标存在问题，即评估过程仍然严重依赖于零样本或交叉数据集流水线上的闭集指标，而未考虑预测和实际类别之间的相似性。为了解决这个问题，我们首先通过综合定量分析和用户研究使用 WordNet 语言统计、文本嵌入和语言模型调查了 11 种不同的相似性测量方法。在此基础上，我们设计了针对三个开放词汇分割任务的新型评估指标，即 Open mIoU、Open AP 和 Open PQ。我们在三个分割任务的 12 种开放词汇方法上对所提出的评估指标进行了基准测试。尽管相似性距离的相对主观性，我们证明了我们的指标仍然能够很好地评估现有的开放词汇分割方法的开放能力。我们希望我们的工作能够为社区带来关于如何评估模型的开放能力的新思考。评估代码在 github 上发布。