An engaging and provocative question can open up a great conversation. In
this work, we explore a novel scenario: a conversation agent views a set of the
user's photos (for example, from social media platforms) and asks an engaging
question to initiate a conversation with the user. The existing
vision-to-question models mostly generate tedious and obvious questions, which
might not be ideals conversation starters. This paper introduces a two-phase
framework that first generates a visual story for the photo set and then uses
the story to produce an interesting question. The human evaluation shows that
our framework generates more response-provoking questions for starting
conversations than other vision-to-question baselines.

该论文提出了一种基于视觉故事的两阶段框架，用于生成启发性问题，经人工评估，其产生的问题比其他基线模型更容易引起交流，成为理想的交流启动器。

借助对话性视觉问题生成展开谈话

Let's Talk! Striking Up Conversations via Conversational Visual Question Generation

While much research has been done in text-to-image synthesis, little work has
been done to explore the usage of linguistic structure of the input text. Such
information is even more important for story visualization since its inputs
have an explicit narrative structure that needs to be translated into an image
sequence (or visual story). Prior work in this domain has shown that there is
ample room for improvement in the generated image sequence in terms of visual
quality, consistency and relevance. In this paper, we first explore the use of
constituency parse trees using a Transformer-based recurrent architecture for
encoding structured input. Second, we augment the structured input with
commonsense information and study the impact of this external knowledge on the
generation of visual story. Third, we also incorporate visual structure via
bounding boxes and dense captioning to provide feedback about the
characters/objects in generated images within a dual learning setup. We show
that off-the-shelf dense-captioning models trained on Visual Genome can improve
the spatial structure of images from a different target domain without needing
fine-tuning. We train the model end-to-end using intra-story contrastive loss
(between words and image sub-regions) and show significant improvements in
several metrics (and human evaluation) for multiple datasets. Finally, we
provide an analysis of the linguistic and visuo-spatial information. Code and
data: this https URL

本文讨论如何更好地将文本转化为对应的图像序列，包括利用 Transformers、constituency parse trees、commonsense 信息以及 visuo-spatial 信息，最终提高了生成图像的质量和一致性。