Creation of images using generative adversarial networks has been widely
adapted into multi-modal regime with the advent of multi-modal representation
models pre-trained on large corpus. Various modalities sharing a common
representation space could be utilized to guide the generative models to create
images from text or even from audio source. Departing from the previous methods
that solely rely on either text or audio, we exploit the expressiveness of both
modality. Based on the fusion of text and audio, we create video whose content
is consistent with the distinct modalities that are provided. A simple approach
to automatically segment the video into variable length intervals and maintain
time consistency in generated video is part of our method. Our proposed
framework for generating music video shows promising results in application
level where users can interactively feed in music source and text source to
create artistic music videos. Our code is available at
this https URL

利用多模态特征空间中的文本和音频嵌入指导生成模型的生成图像，进一步实现音乐视频创作，我们提出的方法需要对视频进行自动分段，同时保持时间上的一致性，与以前的方法不同，我们的方法融合了文本和音频两个模态，展示了良好的应用效果