Continuous-output neural machine translation (CoNMT) replaces the discrete
next-word prediction problem with an embedding prediction. The semantic
structure of the target embedding space (i.e., closeness of related words) is
intuitively believed to be crucial. We challenge this assumption and show that
completely random output embeddings can outperform laboriously pretrained ones,
especially on larger datasets. Further investigation shows this surprising
effect is strongest for rare words, due to the geometry of their embeddings. We
shed further light on this finding by designing a mixed strategy that combines
random and pre-trained embeddings for different tokens.

连续输出神经机器翻译（CoNMT）通过嵌入预测替代了离散的下一个词预测问题；我们挑战了相关单词间语义结构（即相关单词的接近度）对此的重要性的假设，并展示了完全随机的输出嵌入能够优于经过艰苦预训练的嵌入，尤其在更大的数据集上，对于罕见词而言这种令人惊讶的效果最为显著，这是由于他们嵌入的几何形状造成的；我们通过设计一种混合策略来进一步探究此发现，该策略将随机和预训练嵌入用于不同的标记。