Self-supervised speech (SSL) models have recently become widely adopted for
many downstream speech processing tasks. The general usage pattern is to employ
SSL models as feature extractors, and then train a downstream prediction head
to solve a specific task. However, different layers of SSL models have been
shown to capture different types of information, and the methods of combining
them are not well studied. To this end, we extend the general framework for SSL
model utilization by proposing the interface that connects the upstream and
downstream. Under this view, the dominant technique of combining features via a
layerwise weighted sum can be regarded as a specific interface. We propose
several alternative interface designs and demonstrate that the weighted sum
interface is suboptimal for many tasks. In particular, we show that a
convolutional interface whose depth scales logarithmically with the depth of
the upstream model consistently outperforms many other interface designs.

提出了一种连接上游和下游模型的接口，通过使用卷积接口设计，其深度与上游模型的深度按对数比例计算，在许多任务上始终优于其他接口设计。

自我监督语音模型界面设计

Interface Design for Self-Supervised Speech Models

For self-supervised speech processing, it is crucial to use pretrained models
as speech representation extractors. In recent works, increasing the size of
the model has been utilized in acoustic model training in order to achieve
better performance. In this paper, we propose Audio ALBERT, a lite version of
the self-supervised speech representation model. We use the representations
with two downstream tasks, speaker identification, and phoneme classification.
We show that Audio ALBERT is capable of achieving competitive performance with
those huge models in the downstream tasks while utilizing 91\% fewer
parameters. Moreover, we use some simple probing models to measure how much the
information of the speaker and phoneme is encoded in latent representations. In
probing experiments, we find that the latent representations encode richer
information of both phoneme and speaker than that of the last layer.

本文提出了一种自监督的语音表示模型 Audio ALBERT，比已有模型使用更少的参数，在说话人识别和音素分类等下游任务中实现了竞争性的性能，并发现隐表示编码了更丰富的说话人和音素信息。