Despite the remarkable progress of image captioning, existing captioners
typically lack the controllable capability to generate desired image captions,
e.g., describing the image in a rough or detailed manner, in a factual or
emotional view, etc. In this paper, we show that a unified model is qualified
to perform well in diverse domains and freely switch among multiple styles.
Such a controllable capability is achieved by embedding the prompt learning
into the image captioning framework. To be specific, we design a set of prompts
to fine-tune the pre-trained image captioner. These prompts allow the model to
absorb stylized data from different domains for joint training, without
performance degradation in each domain. Furthermore, we optimize the prompts
with learnable vectors in the continuous word embedding space, avoiding the
heuristic prompt engineering and meanwhile exhibiting superior performance. In
the inference stage, our model is able to generate desired stylized captions by
choosing the corresponding prompts. Extensive experiments verify the
controllable capability of the proposed method. Notably, we achieve outstanding
performance on two diverse image captioning benchmarks including COCO Karpathy
split and TextCaps using a unified model.

本文提出了一种统一模型，能够在不同领域间自由切换并实现生成所需风格的图像描述，这种可控能力通过在图像描述的框架中嵌入提示学习来实现，具体采用一组提示用于微调预训练的图像描述器，而这些提示则在连续的词嵌入空间中通过可学习向量进行优化，从而避免启发式提示工程，并展现出卓越的性能。

通过提示实现可控的图像描述

Controllable Image Captioning via Prompting

Stylized image captioning systems aim to generate a caption not only
semantically related to a given image but also consistent with a given style
description. One of the biggest challenges with this task is the lack of
sufficient paired stylized data. Many studies focus on unsupervised approaches,
without considering from the perspective of data augmentation. We begin with
the observation that people may recall similar emotions when they are in
similar scenes, and often express similar emotions with similar style phrases,
which underpins our data augmentation idea. In this paper, we propose a novel
Extract-Retrieve-Generate data augmentation framework to extract style phrases
from small-scale stylized sentences and graft them to large-scale factual
captions. First, we design the emotional signal extractor to extract style
phrases from small-scale stylized sentences. Second, we construct the plugable
multi-modal scene retriever to retrieve scenes represented with pairs of an
image and its stylized caption, which are similar to the query image or caption
in the large-scale factual data. In the end, based on the style phrases of
similar scenes and the factual description of the current scene, we build the
emotion-aware caption generator to generate fluent and diversified stylized
captions for the current scene. Extensive experimental results show that our
framework can alleviate the data scarcity problem effectively. It also
significantly boosts the performance of several existing image captioning
models in both supervised and unsupervised settings, which outperforms the
state-of-the-art stylized image captioning methods in terms of both sentence
relevance and stylishness by a substantial margin.

该论文提出了一种新的数据扩增框架来实现语义相关、风格一致的图像描述生成，在以往仅使用无监督学习方法来进行风格生成的基础上，利用情感词组从小规模缩影句子中提取风格语言，将其与大规模事实描述相结合，采用多模式场景检索和情感感知式生成器生成一系列分别各异、口语化的描述。实验证明，该方法能够有效缓解数据稀缺的问题，同时在句子相关性和时尚性等方面都显着超越了现有技术方法的最新水平。