Artificial Intelligence makes great advances today and starts to bridge the
gap between vision and language. However, we are still far from understanding,
explaining and controlling explicitly the visual content from a linguistic
perspective, because we still lack a common explainable representation between
the two domains. In this work we come to address this limitation and propose
the Graph of Events in Space and Time (GEST), by which we can represent, create
and explain, both visual and linguistic stories. We provide a theoretical
justification of our model and an experimental validation, which proves that
GEST can bring a solid complementary value along powerful deep learning models.
In particular, GEST can help improve at the content-level the generation of
videos from text, by being easily incorporated into our novel video generation
engine. Additionally, by using efficient graph matching techniques, the GEST
graphs can also improve the comparisons between texts at the semantic level.

人工智能在填补视觉与语言之间的差距方面取得了重大进展，但我们仍然远未从语言的角度理解、解释和显式地控制视觉内容，因为两个领域之间仍然缺乏一个共同的可解释表示。在本研究中，我们针对这个限制提出了 “时空事件图（GEST）”，通过它我们可以表示、创建和解释视觉和语言故事。我们提供了我们模型的理论证明和实验验证，证明 GEST 能为强大的深度学习模型带来有实质性的补充价值。特别是，GEST 可以在内容层面上提高从文本生成视频的能力，通过轻松地融入我们的创新视频生成引擎。此外，通过使用高效的图匹配技术，GEST 图还可以改进语义层面上的文本比较。