Image-guided story ending generation (IgSEG) is to generate a story ending
based on given story plots and ending image. Existing methods focus on
cross-modal feature fusion but overlook reasoning and mining implicit
information from story plots and ending image. To tackle this drawback, we
propose a multimodal event transformer, an event-based reasoning framework for
IgSEG. Specifically, we construct visual and semantic event graphs from story
plots and ending image, and leverage event-based reasoning to reason and mine
implicit information in a single modality. Next, we connect visual and semantic
event graphs and utilize cross-modal fusion to integrate different-modality
features. In addition, we propose a multimodal injector to adaptive pass
essential information to decoder. Besides, we present an incoherence detection
to enhance the understanding context of a story plot and the robustness of
graph modeling for our model. Experimental results show that our method
achieves state-of-the-art performance for the image-guided story ending
generation.

提出了一种基于多模态事件转换器的图像引导故事结尾生成方法，该方法利用事件图、跨模态融合和事件推理等技术从故事情节和结尾图像中推导隐含信息，并在解码器中适应性地注入必要信息，实验证明其在故事结尾生成方面性能优于现有方法。