Visual Story-Telling is the process of forming a multi-sentence story from a
set of images. Appropriately including visual variation and contextual
information captured inside the input images is one of the most challenging
aspects of visual storytelling. Consequently, stories developed from a set of
images often lack cohesiveness, relevance, and semantic relationship. In this
paper, we propose a novel Vision Transformer Based Model for describing a set
of images as a story. The proposed method extracts the distinct features of the
input images using a Vision Transformer (ViT). Firstly, input images are
divided into 16X16 patches and bundled into a linear projection of flattened
patches. The transformation from a single image to multiple image patches
captures the visual variety of the input visual patterns. These features are
used as input to a Bidirectional-LSTM which is part of the sequence encoder.
This captures the past and future image context of all image patches. Then, an
attention mechanism is implemented and used to increase the discriminatory
capacity of the data fed into the language model, i.e. a Mogrifier-LSTM. The
performance of our proposed model is evaluated using the Visual Story-Telling
dataset (VIST), and the results show that our model outperforms the current
state of the art models.

该研究提出了一种基于 Vision Transformer 的模型来描述图像集，借助双向 LSTM 和注意力机制来捕捉图像的上下文信息和语义关系，模型在 Visual Story-Telling 数据集上的表现明显超过了目前的最先进模型。