For an image with multiple scene texts, different people may be interested in
different text information. Current text-aware image captioning models are not
able to generate distinctive captions according to various information needs.
To explore how to generate personalized text-aware captions, we define a new
challenging task, namely Question-controlled Text-aware Image Captioning
(Qc-TextCap). With questions as control signals, this task requires models to
understand questions, find related scene texts and describe them together with
objects fluently in human language. Based on two existing text-aware captioning
datasets, we automatically construct two datasets, ControlTextCaps and
ControlVizWiz to support the task. We propose a novel Geometry and Question
Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to
fuse region-level object features and region-level scene text features with
considering spatial relationships. Then, we design a Question-guided Encoder to
select the most relevant visual features for each question. Finally, GQAM
generates a personalized text-aware caption with a Multimodal Decoder. Our
model achieves better captioning performance and question answering ability
than carefully designed baselines on both two datasets. With questions as
control signals, our model generates more informative and diverse captions than
the state-of-the-art text-aware captioning model. Our code and datasets are
publicly available at this https URL

本研究提出了一种新的受控图像文本生成任务 Qc-TextCap，并通过设计问题控制模型（GQAM）达到比现有模型更好的文本生成性能和问题答案能力，同时构建了两个数据集（ControlTextCaps 和 ControlVizWiz），提高了多样化和信息性。