Data-driven unit discovery in self-supervised learning (SSL) of speech has
embarked on a new era of spoken language processing. Yet, the discovered units
often remain in phonetic space, limiting the utility of SSL representations.
Here, we demonstrate that a syllabic organization emerges in learning
sentence-level representation of speech. In particular, we adopt
"self-distillation" objective to fine-tune the pretrained HuBERT with an
aggregator token that summarizes the entire sentence. Without any supervision,
the resulting model draws definite boundaries in speech, and the
representations across frames show salient syllabic structures. We demonstrate
that this emergent structure largely corresponds to the ground truth syllables.
Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating
sentence-level representation of speech. When compared to previous models, our
model outperforms in both unsupervised syllable discovery and learning
sentence-level representation. Together, we demonstrate that the
self-distillation of HuBERT gives rise to syllabic organization without relying
on external labels or modalities, and potentially provides novel data-driven
units for spoken language modeling.

在自监督学习中，通过基于数据的单元发现在语音处理领域迈入了新的阶段。本文展示了在学习语音的句子级表示中出现了音节组织。通过采用 “自蒸馏” 目标函数，在没有任何监督的情况下，对预训练的 HuBERT 进行微调，并添加一个汇总整个句子的聚合器标记。实验结果表明，该模型在语音中画定了明确的边界，并且帧之间的表示显示出显著的音节结构。此外，我们提出了一个用于评估语音句子级表示的新的基准任务，即 “Spoken Speech ABX”。与先前的模型相比，我们的模型在无监督音节发现和学习句子级表示方面表现出色。我们展示了 HuBERT 的自蒸馏能够产生音节组织，而不依赖外部标签或模态，并可能为口语语言建模提供新的、基于数据的单元。