We propose an approach to build a neural machine translation system with no
supervised resources (i.e., no parallel corpora) using multimodal embedded
representation over texts and images. Based on the assumption that text
documents are often likely to be described with other multimedia information
(e.g., images) somewhat related to the content, we try to indirectly estimate
the relevance between two languages. Using multimedia as the "pivot", we
project all modalities into one common hidden space where samples belonging to
similar semantic concepts should come close to each other, whatever the
observed space of each sample is. This modality-agnostic representation is the
key to bridging the gap between different modalities. Putting a decoder on top
of it, our network can flexibly draw the outputs from any input modality.
Notably, in the testing phase, we need only source language texts as the input
for translation. In experiments, we tested our method on two benchmarks to show
that it can achieve reasonable translation performance. We compared and
investigated several possible implementations and found that an end-to-end
model that simultaneously optimized both rank loss in multimodal encoders and
cross-entropy loss in decoders performed the best.

使用多模式嵌入表示法构建神经机器翻译系统，在没有平行语料库的情况下，通过图像和文本信息之间的相关性间接估计两种语言之间的相关度，并在最终测试阶段中，只需要源语言的文本作为翻译输入。在实验中，我们使用端到端模型优化多模态编码器中的排序损失和解码器中的交叉熵损失，取得了不错的翻译结果。