Existing studies on self-supervised speech representation learning have
focused on developing new training methods and applying pre-trained models for
different applications. However, the quality of these models is often measured
by the performance of different downstream tasks. How well the representations
access the information of interest is less studied. In this work, we take a
closer look into existing self-supervised methods of speech from an
information-theoretic perspective. We aim to develop metrics using mutual
information to help practical problems such as model design and selection. We
use linear probes to estimate the mutual information between the target
information and learned representations, showing another insight into the
accessibility to the target information from speech representations. Further,
we explore the potential of evaluating representations in a self-supervised
fashion, where we estimate the mutual information between different parts of
the data without using any labels. Finally, we show that both supervised and
unsupervised measures echo the performance of the models on layer-wise linear
probing and speech recognition.

从信息论的角度出发，了解自监督语音表示学习的现有方法，通过使用互信息来开发度量方法，从而帮助模型设计和选择，并通过线性探测来评估表示的可访问性，展示模型在层次线性探测和语音识别上的性能。