We investigate deep generative models that can exchange multiple modalities
bi-directionally, e.g., generating images from corresponding texts and vice
versa. Recently, some studies handle multiple modalities on deep generative
models, such as variational autoencoders (VAEs). However, these models
typically assume that modalities are forced to have a conditioned relation,
i.e., we can only generate modalities in one direction. To achieve our
objective, we should extract a joint representation that captures high-level
concepts among all modalities and through which we can exchange them
bi-directionally. As described herein, we propose a joint multimodal
variational autoencoder (JMVAE), in which all modalities are independently
conditioned on joint representation. In other words, it models a joint
distribution of modalities. Furthermore, to be able to generate missing
modalities from the remaining modalities properly, we develop an additional
method, JMVAE-kl, that is trained by reducing the divergence between JMVAE's
encoder and prepared networks of respective modalities. Our experiments show
that our proposed method can obtain appropriate joint representation from
multiple modalities and that it can generate and reconstruct them more properly
than conventional VAEs. We further demonstrate that JMVAE can generate multiple
modalities bi-directionally.

本文介绍了一种基于深度生成模型的多模态交换方法，提出了条件独立的联合表示的概念，实现了对不同模态之间的高级概念进行双向交换，并通过对比实验表明该模型的有效性。

深度生成模型的联合多模态学习

Joint Multimodal Learning with Deep Generative Models

Deep neural networks continue to advance the state-of-the-art of image
recognition tasks with various methods. However, applications of these methods
to multimodality remain limited. We present Multimodal Residual Networks (MRN)
for the multimodal residual learning of visual question-answering, which
extends the idea of the deep residual learning. Unlike the deep residual
learning, MRN effectively learns the joint representation from vision and
language information. The main idea is to use element-wise multiplication for
the joint residual mappings exploiting the residual learning of the attentional
models in recent studies. Various alternative models introduced by
multimodality are explored based on our study. We achieve the state-of-the-art
results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks.
Moreover, we introduce a novel method to visualize the attention effect of the
joint representations for each learning block using back-propagation algorithm,
even though the visual features are collapsed without spatial information.

介绍了 Multimodal Residual Networks（MRN），一种通过联合残差映射学习视觉问答的多模态系统，利用注意力模型的残差学习来将视觉和语言信息联合表示，获得了 Visual QA 数据集开放式和多项选择任务的最先进结果，并通过反向传播算法引入了一种可视化联合表示的注意力效果的新方法。