Neural language models learn word representations, or embeddings, that
capture rich linguistic and conceptual information. Here we investigate the
embeddings learned by neural machine translation models, a recently-developed
class of neural language model. We show that embeddings from translation models
outperform those learned by monolingual models at tasks that require knowledge
of both conceptual similarity and lexical-syntactic role. We further show that
these effects hold when translating from both English to French and English to
German, and argue that the desirable properties of translation embeddings
should emerge largely independently of the source and target languages.
Finally, we apply a new method for training neural translation models with very
large vocabularies, and show that this vocabulary expansion algorithm results
in minimal degradation of embedding quality. Our embedding spaces can be
queried in an online demo and downloaded from our web page. Overall, our
analyses indicate that translation-based embeddings should be used in
applications that require concepts to be organised according to similarity
and/or lexical function, while monolingual embeddings are better suited to
modelling (nonspecific) inter-word relatedness.

该研究探讨神经机器翻译模型所学到的嵌入，在需要同时考虑概念相似性和词汇 - 句法角色知识的任务中，它们的性能优于单语言模型所学到的嵌入。研究结果还表明，词汇扩展算法对嵌入质量的影响很小。