Many self-supervised speech models (S3Ms) have been introduced over the last
few years, producing performance and data efficiency improvements for a variety
of speech tasks. Evidence is emerging that different S3Ms encode linguistic
information in different layers, and also that some S3Ms appear to learn
phone-like sub-word units. However, the extent to which these models capture
larger linguistic units, such as words, and where word-related information is
encoded, remains unclear. In this study, we conduct several analyses of word
segment representations extracted from different layers of three S3Ms:
wav2vec2, HuBERT, and WavLM. We employ canonical correlation analysis (CCA), a
lightweight analysis tool, to measure the similarity between these
representations and word-level linguistic properties. We find that the maximal
word-level linguistic content tends to be found in intermediate model layers,
while some lower-level information like pronunciation is also retained in
higher layers of HuBERT and WavLM. Syntactic and semantic word attributes have
similar layer-wise behavior. We also find that, for all of the models tested,
word identity information is concentrated near the center of each word segment.
We then test the layer-wise performance of the same models, when used directly
with no additional learned parameters, on several tasks: acoustic word
discrimination, word segmentation, and semantic sentence similarity. We find
similar layer-wise trends in performance, and furthermore, find that when using
the best-performing layer of HuBERT or WavLM, it is possible to achieve
performance on word segmentation and sentence similarity that rivals more
complex existing approaches.

本研究发现，不同的自监督语音模型可以在不同的层次编码语言特征，在中间层最大程度地捕获了词级的信息，同时在较高层保留了发音等低层次信息，并用在无额外参数的情况下测试了这些模型的层次表现，同时发现使用 HuBERT 或 WavLM 的最佳表现层可以实现与更复杂的方法相媲美的词分割和语义句子相似性的表现。