End-to-end optimization has achieved state-of-the-art performance on many
specific problems, but there is no straight-forward way to combine pretrained
models for new problems. Here, we explore improving modularity by learning a
post-hoc interface between two existing models to solve a new task.
Specifically, we take inspiration from neural machine translation, and cast the
challenging problem of cross-modal domain transfer as unsupervised translation
between the latent spaces of pretrained deep generative models. By abstracting
away the data representation, we demonstrate that it is possible to transfer
across different modalities (e.g., image-to-audio) and even different types of
generative models (e.g., VAE-to-GAN). We compare to state-of-the-art techniques
and find that a straight-forward variational autoencoder is able to best bridge
the two generative models through learning a shared latent space. We can
further impose supervised alignment of attributes in both domains with a
classifier in the shared latent space. Through qualitative and quantitative
evaluations, we demonstrate that locality and semantic alignment are preserved
through the transfer process, as indicated by high transfer accuracies and
smooth interpolations within a class. Finally, we show this modular structure
speeds up training of new interface models by several orders of magnitude by
decoupling it from expensive retraining of base generative models.

本篇论文提出了一种基于先前训练的深度生成模型，利用潜变量空间进行无监督跨模态域迁移的新方法，并探索了通过学习一个后续接口来提高模块性的可能性。通过定量和定性的实验证明了该方法的有效性，证明了在迁移过程中局部性和语义对齐得到了保留，而且通过这种模块化结构可以大大加快新接口模型的训练速度。