Visual and audio modalities are two symbiotic modalities underlying videos,
which contain both common and complementary information. If they can be mined
and fused sufficiently, performances of related video tasks can be
significantly enhanced. However, due to the environmental interference or
sensor fault, sometimes, only one modality exists while the other is abandoned
or missing. By recovering the missing modality from the existing one based on
the common information shared between them and the prior information of the
specific modality, great bonus will be gained for various vision tasks. In this
paper, we propose a Cross-Modal Cycle Generative Adversarial Network (CMCGAN)
to handle cross-modal visual-audio mutual generation. Specifically, CMCGAN is
composed of four kinds of subnetworks: audio-to-visual, visual-to-audio,
audio-to-audio and visual-to-visual subnetworks respectively, which are
organized in a cycle architecture. CMCGAN has several remarkable advantages.
Firstly, CMCGAN unifies visual-audio mutual generation into a common framework
by a joint corresponding adversarial loss. Secondly, through introducing a
latent vector with Gaussian distribution, CMCGAN can handle dimension and
structure asymmetry over visual and audio modalities effectively. Thirdly,
CMCGAN can be trained end-to-end to achieve better convenience. Benefiting from
CMCGAN, we develop a dynamic multimodal classification network to handle the
modality missing problem. Abundant experiments have been conducted and validate
that CMCGAN obtains the state-of-the-art cross-modal visual-audio generation
results. Furthermore, it is shown that the generated modality achieves
comparable effects with those of original modality, which demonstrates the
effectiveness and advantages of our proposed method.

本文提出了一个基于交叉模态循环生成对抗网络的视听互生成模型，它能有效地从一种模态中恢复另一种模态，从而用于动态多模态分类任务，并通过丰富的实验验证表明该方法具有很好的生成效果。