In recent years, there has been a growing interest in Semantic Image Synthesis (SIS) through the use of Generative Adversarial Networks (GANs) and diffusion models. This field has seen innovations such as the implementation of specialized loss functions tailored for this task, diverging from the more general approaches in Image-to-Image (I2I) translation. While the concept of Semantic Video Synthesis (SVS)$\unicode{x2013}$the generation of temporally coherent, realistic sequences of images from semantic maps$\unicode{x2013}$is newly formalized in this paper, some existing methods have already explored aspects of this field. Most of these approaches rely on generic loss functions designed for video-to-video translation or require additional data to achieve temporal coherence. In this paper, we introduce the SVS-GAN, a framework specifically designed for SVS, featuring a custom architecture and loss functions. Our approach includes a triple-pyramid generator that utilizes SPADE blocks. Additionally, we employ a U-Net-based network for the image discriminator, which performs semantic segmentation for the OASIS loss. Through this combination of tailored architecture and objective engineering, our framework aims to bridge the existing gap between SIS and SVS, outperforming current state-of-the-art models on datasets like Cityscapes and KITTI-360.

本研究解决了语义视频合成（SVS）领域现有方法依赖通用损失函数和大数据集的问题。我们提出的SVS-GAN框架采用特定的网络架构和损失函数，通过三层金字塔生成器和基于U-Net的图像判别器，显著提升了OASIS损失的语义分割能力。研究结果表明，该方法在Cityscapes和KITTI-360等数据集上超越了现有的最先进模型，展示了SVS在图像到视频合成中的潜力。

SVS-GAN：利用GAN进行语义视频合成