Story visualization (SV) is a challenging text-to-image generation task for
the difficulty of not only rendering visual details from the text descriptions
but also encoding a long-term context across multiple sentences. While prior
efforts mostly focus on generating a semantically relevant image for each
sentence, encoding a context spread across the given paragraph to generate
contextually convincing images (e.g., with a correct character or with a proper
background of the scene) remains a challenge. To this end, we propose a novel
memory architecture for the Bi-directional Transformers with an online text
augmentation that generates multiple pseudo-descriptions as supplementary
supervision during training, for better generalization to the language
variation at inference. In extensive experiments on the two popular SV
benchmarks, i.e., the Pororo-SV and Flintstones-SV, the proposed method
significantly outperforms the state of the arts in various evaluation metrics
including FID, character F1, frame accuracy, BLEU-2/3, and R-precision with
similar or less computational complexity.

故事可视化是一项具有挑战性的文本到图像生成任务，不仅需要从文本描述中呈现视觉细节，还需要对跨多个句子的长期上下文进行编码。本文提出了一种新颖的记忆架构用于双向 Transformer，并采用在线文本增强生成多个伪描述作为训练过程中的辅助监督，以更好地适应推理中的语言变化，通过对 Pororo-SV 和 Flintstones-SV 两个流行的 SV 基准测试的广泛实验，该方法在各种评估指标上明显优于现有技术，并具有相似或更少的计算复杂性。