In this paper, we investigate Open-Vocabulary 3D Instance Segmentation
(OV-3DIS) with free-form language instructions. Earlier works that rely on only
annotated base categories for training suffer from limited generalization to
unseen novel categories. Recent works mitigate poor generalizability to novel
categories by generating class-agnostic masks or projecting generalized masks
from 2D to 3D, but disregard semantic or geometry information, leading to
sub-optimal performance. Instead, generating generalizable but semantic-related
masks directly from 3D point clouds would result in superior outcomes. In this
paper, we introduce Segment any 3D Object with LanguagE (SOLE), which is a
semantic and geometric-aware visual-language learning framework with strong
generalizability by generating semantic-related masks directly from 3D point
clouds. Specifically, we propose a multimodal fusion network to incorporate
multimodal semantics in both backbone and decoder. In addition, to align the 3D
segmentation model with various language instructions and enhance the mask
quality, we introduce three types of multimodal associations as supervision.
Our SOLE outperforms previous methods by a large margin on ScanNetv2,
ScanNet200, and Replica benchmarks, and the results are even close to the
fully-supervised counterpart despite the absence of class annotations in the
training. Furthermore, extensive qualitative results demonstrate the
versatility of our SOLE to language instructions.

本文介绍了一种基于语言指令的语义和几何感知的视觉语言学习框架，通过直接从 3D 点云生成语义相关的掩码，从而在 ScanNetv2、ScanNet200 和 Replica 基准测试中明显优于现有方法，且结果接近全监督对照组，尽管训练数据中缺乏类别标注。

使用语言对任意 3D 物体进行分割

Segment Any 3D Object with Language

Recently, visual-language learning has shown great potential in enhancing
visual-based person re-identification (ReID). Existing visual-language
learning-based ReID methods often focus on whole-body scale image-text feature
alignment, while neglecting supervisions on fine-grained part features. This
choice simplifies the learning process but cannot guarantee within-part feature
semantic consistency thus hindering the final performance. Therefore, we
propose to enhance fine-grained visual features with part-informed language
supervision for ReID tasks. The proposed method, named Part-Informed
Visual-language Learning ($\pi$-VL), suggests that (i) a human parsing-guided
prompt tuning strategy and (ii) a hierarchical fusion-based visual-language
alignment paradigm play essential roles in ensuring within-part feature
semantic consistency. Specifically, we combine both identity labels and parsing
maps to constitute pixel-level text prompts and fuse multi-stage visual
features with a light-weight auxiliary head to perform fine-grained image-text
alignment. As a plug-and-play and inference-free solution, our $\pi$-VL
achieves substantial improvements over previous state-of-the-arts on four
common-used ReID benchmarks, especially reporting 90.3% Rank-1 and 76.5% mAP
for the most challenging MSMT17 database without bells and whistles.

最近，视觉语言学习在增强基于视觉的人物再识别方面展示了巨大的潜力。本文提出了一种名为 Part-Informed Visual-language Learning (π-VL) 的方法，通过在 ReID 任务中利用部分信息语言监督来增强细粒度的视觉特征，该方法在四个常用的 ReID 基准测试中取得了显著的改进，特别是在最具挑战性的 MSMT17 数据库中达到了 90.3% 的 Rank-1 和 76.5% 的 mAP，而无需任何额外的技术手段。