Open-vocabulary semantic segmentation is a challenging task that requires
segmenting novel object categories at inference time. Recent works explore
vision-language pre-training to handle this task, but suffer from unrealistic
assumptions in practical scenarios, i.e., low-quality textual category names.
For example, this paradigm assumes that new textual categories will be
accurately and completely provided, and exist in lexicons during pre-training.
However, exceptions often happen when meet with ambiguity for brief or
incomplete names, new words that are not present in the pre-trained lexicons,
and difficult-to-describe categories for users. To address these issues, this
work proposes a novel decomposition-aggregation framework, inspired by human
cognition in understanding new concepts. Specifically, in the decomposition
stage, we decouple class names into diverse attribute descriptions to enrich
semantic contexts. Two attribute construction strategies are designed: using
large language models for common categories, and involving manually labelling
for human-invented categories. In the aggregation stage, we group diverse
attributes into an integrated global description, to form a discriminative
classifier that distinguishes the target object from others. One hierarchical
aggregation is further designed to achieve multi-level alignment and deep
fusion between vision and text. The final result is obtained by computing the
embedding similarity between aggregated attributes and images. To evaluate the
effectiveness, we annotate three datasets with attribute descriptions, and
conduct extensive experiments and ablation studies. The results show the
superior performance of attribute decomposition-aggregation.

本文提出了一种新颖的分解 - 聚合框架，通过属性描述的多样性丰富语义上下文，使用大型语言模型构建常见类别的属性，并通过手动标注设计人工发明类别的属性，将多样属性聚合为一个集成全局描述的判别分类器，进而通过图像和文本之间的层次聚合和深度融合来实现多级对齐，最终通过计算聚合属性和图像之间的嵌入相似度来得到最终结果，实验证明属性分解 - 聚合具有卓越的性能。

基于属性分解聚合的开放词汇语义分割

Open-Vocabulary Semantic Segmentation via Attribute  Decomposition-Aggregation

Text-based person retrieval aims to find the query person based on a textual
description. The key is to learn a common latent space mapping between
visual-textual modalities. To achieve this goal, existing works employ
segmentation to obtain explicitly cross-modal alignments or utilize attention
to explore salient alignments. These methods have two shortcomings: 1) Labeling
cross-modal alignments are time-consuming. 2) Attention methods can explore
salient cross-modal alignments but may ignore some subtle and valuable pairs.
To relieve these issues, we introduce an Implicit Visual-Textual (IVT)
framework for text-based person retrieval. Different from previous models, IVT
utilizes a single network to learn representation for both modalities, which
contributes to the visual-textual interaction. To explore the fine-grained
alignment, we further propose two implicit semantic alignment paradigms:
multi-level alignment (MLA) and bidirectional mask modeling (BMM). The MLA
module explores finer matching at sentence, phrase, and word levels, while the
BMM module aims to mine \textbf{more} semantic alignments between visual and
textual modalities. Extensive experiments are carried out to evaluate the
proposed IVT on public datasets, i.e., CUHK-PEDES, RSTPReID, and ICFG-PEDES.
Even without explicit body part alignment, our approach still achieves
state-of-the-art performance. Code is available at:
this https URL

本研究提出了一种隐式视觉 - 文本框架，通过多级和双向遮罩建模方法实现跨模态匹配，提高了人物的文本检索效果。