Human speakers encode information into raw speech which is then decoded by
the listeners. This complex relationship between encoding (production) and
decoding (perception) is often modeled separately. Here, we test how encoding
and decoding of lexical semantic information can emerge automatically from raw
speech in unsupervised generative deep convolutional networks that combine the
production and perception principles of speech. We introduce, to our knowledge,
the most challenging objective in unsupervised lexical learning: a network that
must learn unique representations for lexical items with no direct access to
training data. We train several models (ciwGAN and fiwGAN arXiv:2006.02951) and
test how the networks classify acoustic lexical items in unobserved test data.
Strong evidence in favor of lexical learning and a causal relationship between
latent codes and meaningful sublexical units emerge. The architecture that
combines the production and perception principles is thus able to learn to
decode unique information from raw acoustic data without accessing real
training data directly. We propose a technique to explore lexical (holistic)
and sublexical (featural) learned representations in the classifier network.
The results bear implications for unsupervised speech technology, as well as
for unsupervised semantic modeling as language models increasingly bypass text
and operate from raw acoustics.

本文旨在研究如何从原始语音中自动生成词汇语义信息，并将其应用于无人监督的语音技术和语义建模领域。作者建立了包含生产和感知原则的深度卷积神经网络，通过自动编码和分类技术实现词汇语义信息自动解码。研究结果表明该网络能够从原始语音数据中学习到独特的词汇信息，具有良好的应用前景。

同时建模语音识别和合成：将词汇和亚词汇语义信息编码和解码为语音，无需直接访问语音数据

Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data

Most natural language processing tasks require lexical semantic information.
Automated acquisition of this information would thus increase the robustness
and portability of NLP systems. This paper describes an acquisition method
which makes use of fixed correspondences between derivational affixes and
lexical semantic information. One advantage of this method, and of other
methods that rely only on surface characteristics of language, is that the
necessary input is currently available.

本文介绍了一种利用派生词缀和词汇语义信息之间的固定对应关系自动获取词汇语义信息的方法。该方法的优点之一是，与其他仅依赖语言表层特征的方法相比，其必要的输入数据已经可以获取。