Story visualization aims to generate a series of realistic and coherent
images based on a storyline. Current models adopt a frame-by-frame architecture
by transforming the pre-trained text-to-image model into an auto-regressive
manner. Although these models have shown notable progress, there are still
three flaws. 1) The unidirectional generation of auto-regressive manner
restricts the usability in many scenarios. 2) The additional introduced story
history encoders bring an extremely high computational cost. 3) The story
visualization and continuation models are trained and inferred independently,
which is not user-friendly. To these ends, we propose a bidirectional, unified,
and efficient framework, namely StoryImager. The StoryImager enhances the
storyboard generative ability inherited from the pre-trained text-to-image
model for a bidirectional generation. Specifically, we introduce a Target Frame
Masking Strategy to extend and unify different story image generation tasks.
Furthermore, we propose a Frame-Story Cross Attention Module that decomposes
the cross attention for local fidelity and global coherence. Moreover, we
design a Contextual Feature Extractor to extract contextual information from
the whole storyline. The extensive experimental results demonstrate the
excellent performance of our StoryImager. The code is available at
this https URL

StoryImager 是一个双向的、统一的、高效的框架，通过增强继承于预训练的文本到图像模型的 storyboard 生成能力实现双向生成。具体而言，我们引入了一种目标框架掩码策略来扩展和统一不同的故事图像生成任务。此外，我们提出了一种框架 - 故事交叉注意力模块，用于分解交叉注意力以实现局部保真度和全局一致性。此外，我们设计了一个上下文特征提取器，从整个故事线中提取上下文信息。广泛的实验结果证明了我们的 StoryImager 具有出色的性能。