Generating spoken word embeddings that possess semantic information is a
fascinating topic. Compared with text-based embeddings, they cover both
phonetic and semantic characteristics, which can provide richer information and
are potentially helpful for improving ASR and speech translation systems. In
this paper, we review and examine the authenticity of a seminal work in this
field: Speech2Vec. First, a homophone-based inspection method is proposed to
check the speech embeddings released by the author of Speech2Vec. There is no
indication that these embeddings are generated by the Speech2Vec model.
Moreover, through further analysis of the vocabulary composition, we suspect
that a text-based model fabricates these embeddings. Finally, we reproduce the
Speech2Vec model, referring to the official code and optimal settings in the
original paper. Experiments showed that this model failed to learn effective
semantic embeddings. In word similarity benchmarks, it gets a correlation score
of 0.08 in MEN and 0.15 in WS-353-SIM tests, which is over 0.5 lower than those
described in the original paper. Our data and code are available.

本文检验了 Speech2Vec 模型的效果，发现其未能学习到有效的语义嵌入，并怀疑原文中提到的语音词向量是由一个基于文本的模型生成的。

同音字揭示真相：针对 Speech2Vec 的现实检验

Homophone Reveals the Truth: A Reality Check for Speech2Vec

In this paper, we propose a novel deep neural network architecture,
Speech2Vec, for learning fixed-length vector representations of audio segments
excised from a speech corpus, where the vectors contain semantic information
pertaining to the underlying spoken words, and are close to other vectors in
the embedding space if their corresponding underlying spoken words are
semantically similar. The proposed model can be viewed as a speech version of
Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows
the methodology of skipgrams or continuous bag-of-words for training. Learning
word embeddings directly from speech enables Speech2Vec to make use of the
semantic information carried by speech that does not exist in plain text. The
learned word embeddings are evaluated and analyzed on 13 widely used word
similarity benchmarks, and outperform word embeddings learned by Word2Vec from
the transcriptions.

本文提出一种新颖的深度神经网络架构 Speech2Vec，用于从语音语料库中学习固定长度的音频片段矢量表示，其向量包含有关底层口语词汇的语义信息，并且如果它们的对应的底层口语词汇在语义上相似，则在嵌入空间中靠近其他向量。 Speech2Vec 的设计基于 RNN 编码器 - 解码器框架，并借用 skipgrams 或连续词袋的方法进行训练。直接从语音中学习单词嵌入使 Speech2Vec 能够利用语音中的语义信息，在 13 个广泛使用的单词相似性基准测试中评估和分析了学习的单词嵌入，并且胜过了从转录中学习的 Word2Vec 单词嵌入。