Despite the remarkable progress of image captioning, existing captioners
typically lack the controllable capability to generate desired image captions,
e.g., describing the image in a rough or detailed manner, in a factual or
emotional view, etc. In this paper, we show that a unified model is qualified
to perform well in diverse domains and freely switch among multiple styles.
Such a controllable capability is achieved by embedding the prompt learning
into the image captioning framework. To be specific, we design a set of prompts
to fine-tune the pre-trained image captioner. These prompts allow the model to
absorb stylized data from different domains for joint training, without
performance degradation in each domain. Furthermore, we optimize the prompts
with learnable vectors in the continuous word embedding space, avoiding the
heuristic prompt engineering and meanwhile exhibiting superior performance. In
the inference stage, our model is able to generate desired stylized captions by
choosing the corresponding prompts. Extensive experiments verify the
controllable capability of the proposed method. Notably, we achieve outstanding
performance on two diverse image captioning benchmarks including COCO Karpathy
split and TextCaps using a unified model.

本文提出了一种统一模型，能够在不同领域间自由切换并实现生成所需风格的图像描述，这种可控能力通过在图像描述的框架中嵌入提示学习来实现，具体采用一组提示用于微调预训练的图像描述器，而这些提示则在连续的词嵌入空间中通过可学习向量进行优化，从而避免启发式提示工程，并展现出卓越的性能。