Visual storytelling often uses nontypical aspect-ratio images like scroll paintings, comic strips, and panoramas to create an expressive and compelling narrative. While generative AI has achieved great success and shown the potential to reshape the creative industry, it remains a challenge to generate coherent and engaging content with arbitrary size and controllable style, concept, and layout, all of which are essential for visual storytelling. To overcome the shortcomings of previous methods including repetitive content, style inconsistency, and lack of controllability, we propose MagicScroll, a multi-layered, progressive diffusion-based image generation framework with a novel semantic-aware denoising process. The model enables fine-grained control over the generated image on object, scene, and background levels with text, image, and layout conditions. We also establish the first benchmark for nontypical aspect-ratio image generation for visual storytelling including mediums like paintings, comics, and cinematic panoramas, with customized metrics for systematic evaluation. Through comparative and ablation studies, MagicScroll showcases promising results in aligning with the narrative text, improving visual coherence, and engaging the audience. We plan to release the code and benchmark in the hope of a better collaboration between AI researchers and creative practitioners involving visual storytelling.

使用MagicScroll，一种多层、渐进式扩散型图像生成框架，结合新颖的语义感知去噪过程，提供了对于生成图像的对象、场景和背景层面的细粒度控制，借助文本、图像和布局条件，能够改善与叙述文本的一致性，提高视觉连贯性，并吸引观众，并且建立了关于视觉叙事中非典型宽高比图像生成的基准，其中包括绘画作品、漫画和电影全景等媒体，并针对其进行了系统评估。

MagicScroll: 多层语义感知去噪的视觉叙事非典型宽高比图像生成