We propose a method to efficiently equip the Segment Anything Model (SAM)
with the ability to generate regional captions. SAM presents strong
generalizability to segment anything while is short for semantic understanding.
By introducing a lightweight query-based feature mixer, we align the
region-specific features with the embedding space of language models for later
caption generation. As the number of trainable parameters is small (typically
in the order of tens of millions), it costs less computation, less memory
usage, and less communication bandwidth, resulting in both fast and scalable
training. To address the scarcity problem of regional caption data, we propose
to first pre-train our model on objection detection and segmentation tasks. We
call this step weak supervision pretraining since the pre-training data only
contains category names instead of full-sentence descriptions. The weak
supervision pretraining allows us to leverage many publicly available object
detection and segmentation datasets. We conduct extensive experiments to
demonstrate the superiority of our method and validate each design choice. This
work serves as a stepping stone towards scaling up regional captioning data and
sheds light on exploring efficient ways to augment SAM with regional semantics.
The project page, along with the associated code, can be accessed via the
following this https URL

提出了一种方法，能够高效地为 Segment Anything Model (SAM) 添加生成区域描述的功能，并且通过引入轻量级的基于查询的特征混合器使区域特征与语言模型的嵌入空间对齐，以便进行后续的描述生成。该方法具有小的可训练参数数量，计算量少、内存使用少和通信带宽少的特点，因此训练速度快且可扩展。通过先在目标检测和分割任务上进行弱监督预训练以解决区域描述数据稀缺问题。该方法的优越性得到了广泛的实验证明，并对每个设计选择进行了验证。本研究在扩展区域描述数据和探索为 SAM 增加区域语义方面具有重要意义。