Visual storytelling aims to generate a narrative based on a sequence of
images, necessitating both vision-language alignment and coherent story
generation. Most existing solutions predominantly depend on paired image-text
training data, which can be costly to collect and challenging to scale. To
address this, we formulate visual storytelling as a visual-conditioned story
generation problem and propose a text-only training method that separates the
learning of cross-modality alignment and story generation. Our approach
specifically leverages the cross-modality pre-trained CLIP model to integrate
visual control into a story generator, trained exclusively on text data.
Moreover, we devise a training-free visual condition planner that accounts for
the temporal structure of the input image sequence while balancing global and
local visual content. The distinctive advantage of requiring only text data for
training enables our method to learn from external text story data, enhancing
the generalization capability of visual storytelling. We conduct extensive
experiments on the VIST benchmark, showcasing the effectiveness of our approach
in both in-domain and cross-domain settings. Further evaluations on expression
diversity and human assessment underscore the superiority of our method in
terms of informativeness and robustness.

利用跨模态预训练的 CLIP 模型，结合纯文本数据训练的视觉条件故事生成器及不依赖训练的视觉条件规划器，提出了一种仅使用文本数据进行训练的视觉叙事方法，有效提高了视觉叙事的泛化能力。在 VIST 基准上进行的广泛实验以及表达多样性和人工评估的进一步评估结果，都突显了我们方法在信息丰富性和稳健性方面的优越性。