Characters are an important aspect of any storyline and identifying and
including them in descriptions is necessary for story understanding. While
previous work has largely ignored identity and generated captions with someone
(anonymized names), recent work formulates id-aware captioning as a
fill-in-the-blanks (FITB) task, where, given a caption with blanks, the goal is
to predict person id labels. However, to predict captions with ids, a two-stage
approach is required: first predict captions with someone, then fill in
identities. In this work, we present a new single stage approach that can
seamlessly switch between id-aware caption generation or FITB when given a
caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared
auto-regressive decoder that benefits from training with FITB and full-caption
generation objectives, while the encoder can benefit from or disregard captions
with blanks as input. Another challenge with id-aware captioning is the lack of
a metric to capture subtle differences between person ids. To this end, we
introduce iSPICE, a caption evaluation metric that focuses on identity tuples
created through intermediate scene graphs. We evaluate MICap on Large-Scale
Movie Description Challenge (LSMDC), where we show a 4.2% improvement in FITB
accuracy, and a 1-2% bump in classic captioning metrics.

本文介绍了一种新的单阶段方法，可以在给定带空白的字幕时无缝切换到基于身份感知的字幕生成或填空任务。我们使用一个共享自回归解码器的模型（MICap），该模型在 FITB 和全字幕生成目标的训练中受益，而编码器可以根据需要利用或忽略输入中带有空白的字幕。此外，我们还引入了一种名为 iSPICE 的字幕评估指标，该指标专注于通过中间场景图创建的身份元组，对于身份感知的字幕生成存在的一个挑战是缺乏捕捉人物身份之间细微差异的度量标准。最后，我们在大规模电影描述挑战（LSMDC）上评估 MICap，其中显示 FITB 准确性提高了 4.2％，传统字幕生成指标增加了 1-2％。