Visual storytelling aims to generate compelling narratives from image
sequences. Existing models often focus on enhancing the representation of the
image sequence, e.g., with external knowledge sources or advanced graph
structures. Despite recent progress, the stories are often repetitive,
illogical, and lacking in detail. To mitigate these issues, we present a novel
framework which integrates visual representations with pretrained language
models and planning. Our model translates the image sequence into a visual
prefix, a sequence of continuous embeddings which language models can
interpret. It also leverages a sequence of question-answer pairs as a blueprint
plan for selecting salient visual concepts and determining how they should be
assembled into a narrative. Automatic and human evaluation on the VIST
benchmark (Huang et al., 2016) demonstrates that blueprint-based models
generate stories that are more coherent, interesting, and natural compared to
competitive baselines and state-of-the-art systems.

将图像序列转化为可解释的视觉前缀，结合预训练语言模型和规划，利用问题 - 答案对作为蓝图计划选择显著的视觉概念，并确定它们如何组织成故事，生成更连贯、有趣和自然的故事。