We capitalize on large amounts of readily-available, synchronous data to
learn a deep discriminative representations shared across three major natural
modalities: vision, sound and language. By leveraging over a year of sound from
video and millions of sentences paired with images, we jointly train a deep
convolutional network for aligned representation learning. Our experiments
suggest that this representation is useful for several tasks, such as
cross-modal retrieval or transferring classifiers between modalities. Moreover,
although our network is only trained with image+text and image+sound pairs, it
can transfer between text and sound as well, a transfer the network never
observed during training. Visualizations of our representation reveal many
hidden units which automatically emerge to detect concepts, independent of the
modality.

利用大规模、同步的数据，我们进行了深度判别式表示学习，在三个主要的自然模态中共享学习。我们的实验表明，这种表示对于跨模态检索或在模态之间转移分类器非常有用。此外，我们的网络虽然只是采用图像 + 文本和图像 + 声音对进行训练，但也可以在文本和声音之间进行转移学习，这在训练期间网络从未观察到。我们的表征的可视化揭示了许多隐藏的单元，这些单元自动出现来检测概念，独立于模态。