We are creating multimedia contents everyday and everywhere. While automatic
content generation has played a fundamental challenge to multimedia community
for decades, recent advances of deep learning have made this problem feasible.
For example, the Generative Adversarial Networks (GANs) is a rewarding approach
to synthesize images. Nevertheless, it is not trivial when capitalizing on GANs
to generate videos. The difficulty originates from the intrinsic structure
where a video is a sequence of visually coherent and semantically dependent
frames. This motivates us to explore semantic and temporal coherence in
designing GANs to generate videos. In this paper, we present a novel Temporal
GANs conditioning on Captions, namely TGANs-C, in which the input to the
generator network is a concatenation of a latent noise vector and caption
embedding, and then is transformed into a frame sequence with 3D
spatio-temporal convolutions. Unlike the naive discriminator which only judges
pairs as fake or real, our discriminator additionally notes whether the video
matches the correct caption. In particular, the discriminator network consists
of three discriminators: video discriminator classifying realistic videos from
generated ones and optimizes video-caption matching, frame discriminator
discriminating between real and fake frames and aligning frames with the
conditioning caption, and motion discriminator emphasizing the philosophy that
the adjacent frames in the generated videos should be smoothly connected as in
real ones. We qualitatively demonstrate the capability of our TGANs-C to
generate plausible videos conditioning on the given captions on two synthetic
datasets (SBMG and TBMG) and one real-world dataset (MSVD). Moreover,
quantitative experiments on MSVD are performed to validate our proposal via
Generative Adversarial Metric and human study.

本文中，我们提出了一个新颖的基于标题的时序生成对抗网络（TGANs-C）来生成视频，该网络结合了数据的语义和时间上的一致性。我们通过定性和定量实验验证了我们的提议，并展示了在给定标题的情况下生成逼真视频的能力。