Controllable Image Captioning (CIC) -- generating image descriptions
following designated control signals -- has received unprecedented attention
over the last few years. To emulate the human ability in controlling caption
generation, current CIC studies focus exclusively on control signals concerning
objective properties, such as contents of interest or descriptive patterns.
However, we argue that almost all existing objective control signals have
overlooked two indispensable characteristics of an ideal control signal: 1)
Event-compatible: all visual contents referred to in a single sentence should
be compatible with the described activity. 2) Sample-suitable: the control
signals should be suitable for a specific image sample. To this end, we propose
a new control signal for CIC: Verb-specific Semantic Roles (VSR). VSR consists
of a verb and some semantic roles, which represents a targeted activity and the
roles of entities involved in this activity. Given a designated VSR, we first
train a grounded semantic role labeling (GSRL) model to identify and ground all
entities for each role. Then, we propose a semantic structure planner (SSP) to
learn human-like descriptive semantic structures. Lastly, we use a role-shift
captioning model to generate the captions. Extensive experiments and ablations
demonstrate that our framework can achieve better controllability than several
strong baselines on two challenging CIC benchmarks. Besides, we can generate
multi-level diverse captions easily. The code is available at:
this https URL

提出了新的控制信号，基于 Controllable Image Captioning (CIC)，包括顶点特定语义角色 (VSR)，通过基于地面的语义角色标记模型 (GSRL) 训练来确定所有实体对于每个角色，借助语义结构计划器（SSP）学习像人类一样的描述性语义结构，并使用角色转移字幕模型生成字幕。