Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive empirical successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.

本研究使用基于规范相关性分析（CCA）的轻量级分析工具, 考察了多种最近模型的中间表示，发现它们所编码的属性在不同层次上发展，变化与预训练目标的选择有关, 进一步探究了这种分析在下游任务中的实用性, 并发现CCA趋势为选择下游任务感兴趣的层提供了可靠的指导, 单层性能往往与使用所有层相当或更好，这意味着更有效利用预训练模型的启示。

自监督语音模型的分层比较分析