The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
models (VFMs). SAM excels in segmentation tasks across diverse domains, while
CLIP is renowned for its zero-shot recognition capabilities. This paper
presents an in-depth exploration of integrating these two models into a unified
framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
model designed for simultaneous interactive segmentation and recognition,
leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
former adapts SAM's knowledge into the CLIP via distillation and learnable
transformer adapters, while the latter transfers CLIP knowledge into SAM,
enhancing its recognition capabilities. Extensive experiments on various
datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
segmentation and recognition tasks, significantly outperforming the naive
baselines of simply combining SAM and CLIP. Furthermore, aided with image
classification data training, our method can segment and recognize
approximately 22,000 classes.

该论文介绍了将 CLIP 和 Segment Anything Model（SAM）集成到统一框架中的深入研究，提出了基于 SAM 的 Open-Vocabulary SAM 模型，实现了同时交互分割和识别的功能，并通过知识转移模块 SAM2CLIP 和 CLIP2SAM 进行知识转移，显著优于简单组合 SAM 和 CLIP 的基准线。此外，通过图像分类数据训练，我们的方法可以分割和识别大约 22,000 个类别。