In this paper, we propose a controllable dense captioner (ControlCap), which
accommodates user's intention to dense captioning by introducing linguistic
guidance. ControlCap is defined as a multimodal embedding bridging
architecture, which comprises multimodal embedding generation (MEG) module and
bi-directional embedding bridging (BEB) module. While MEG module represents
objects/regions by combining embeddings of detailed information with
context-aware ones, it also endows ControlCap the adaptability to specialized
controls by utilizing them as linguistic guidance. BEB module aligns the
linguistic guidance with visual embeddings through borrowing/returning features
from/to the visual domain and gathering such features to predict text
descriptions. Experiments on Visual Genome and VG-COCO datasets show that
ControlCap respectively outperforms the state-of-the-art methods by 1.5% and
3.7% (mAP). Last but not least, with the capability of converting
region-category pairs to region-text pairs, ControlCap is able to act as a
powerful data engine for dense captioning. Code is available at
this https URL

提出了一种名为 ControlCap 的可控制的密集字幕生成器，通过引入语言指导来适应用户对于密集字幕生成的意图。ControlCap 是由多模态嵌入生成模块和双向嵌入桥接模块组成的多模态嵌入桥接架构。实验证明，ControlCap 在 Visual Genome 和 VG-COCO 数据集上分别比现有方法提高了 1.5% 和 3.7%（mAP）。并且，通过将区域 - 类别对转换为区域 - 文本对，ControlCap 可以作为密集字幕生成的强大数据引擎。