Single-point annotation in visual tasks, with the goal of minimizing
labelling costs, is becoming increasingly prominent in research. Recently,
visual foundation models, such as Segment Anything (SAM), have gained
widespread usage due to their robust zero-shot capabilities and exceptional
annotation performance. However, SAM's class-agnostic output and high
confidence in local segmentation introduce 'semantic ambiguity', posing a
challenge for precise category-specific segmentation. In this paper, we
introduce a cost-effective category-specific segmenter using SAM. To tackle
this challenge, we have devised a Semantic-Aware Instance Segmentation Network
(SAPNet) that integrates Multiple Instance Learning (MIL) with matching
capability and SAM with point prompts. SAPNet strategically selects the most
representative mask proposals generated by SAM to supervise segmentation, with
a specific focus on object category information. Moreover, we introduce the
Point Distance Guidance and Box Mining Strategy to mitigate inherent
challenges: 'group' and 'local' issues in weakly supervised segmentation. These
strategies serve to further enhance the overall segmentation performance. The
experimental results on Pascal VOC and COCO demonstrate the promising
performance of our proposed SAPNet, emphasizing its semantic matching
capabilities and its potential to advance point-prompted instance segmentation.
The code will be made publicly available.

使用 Segment Anything (SAM) 集成 Multiple Instance Learning (MIL) 的 Semantic-Aware Instance Segmentation Network (SAPNet) 有助于解决弱监督分割中 'group' 和 'local' 问题，提高语义匹配能力和基于点提示的实例分割性能。

基于语义感知的点提示实例分割及验证

Semantic-aware SAM for Point-Prompted Instance Segmentation

End-to-end scene text spotting has made significant progress due to its
intrinsic synergy between text detection and recognition. Previous methods
commonly regard manual annotations such as horizontal rectangles, rotated
rectangles, quadrangles,and polygons as a prerequisite, which are much more
expensive than using single-point. For the first time, we demonstrate that
training scene text spotting models can be achieved with an extremely low-cost
single-point annotation by the proposed framework, termed SPTS v2. SPTS v2
reserves the advantage of the auto-regressive Transformer with an Instance
Assignment Decoder (IAD) through sequentially predicting the center points of
all text instances inside the same predicting sequence, while with a Parallel
Recognition Decoder (PRD) for text recognition in parallel. These two decoders
share the same parameters and are interactively connected with a simple but
effective information transmission process to pass the gradient and
information. Comprehensive experiments on various existing benchmark datasets
demonstrate the SPTS v2 can outperform previous state-of-the-art single-point
text spotters with fewer parameters while achieving 14x faster inference speed.
Most importantly, within the scope of our SPTS v2, extensive experiments
further reveal an important phenomenon that single-point serves as the optimal
setting for the scene text spotting compared to non-point, rectangular bounding
box, and polygonal bounding box. Such an attempt provides a significant
opportunity for scene text spotting applications beyond the realms of existing
paradigms. Code is available at this https URL.

本文提出了 SPTS v2 框架，采用单点注释作为代替传统文本检测与识别的昂贵手工注释方法。采用基于自回归 Transformer 的 IAD 对同一预测序列内所有文本实例的中心点进行预测，采用 PRD 进行文本识别。详细实验结果证明该方法可超越现有单点文本定位器的性能而实现 14 倍更快的推断速度。同时，实验还进一步展示了单点注释比非点注释和多边形边界框更为适合场景文本识别。