Typically, unsupervised segmentation of speech into the phone and word-like
units are treated as separate tasks and are often done via different methods
which do not fully leverage the inter-dependence of the two tasks. Here, we
unify them and propose a technique that can jointly perform both, showing that
these two tasks indeed benefit from each other. Recent attempts employ
self-supervised learning, such as contrastive predictive coding (CPC), where
the next frame is predicted given past context. However, CPC only looks at the
audio signal's frame-level structure. We overcome this limitation with a
segmental contrastive predictive coding (SCPC) framework to model the signal
structure at a higher level, e.g., phone level. A convolutional neural network
learns frame-level representation from the raw waveform via noise-contrastive
estimation (NCE). A differentiable boundary detector finds variable-length
segments, which are then used to optimize a segment encoder via NCE to learn
segment representations. The differentiable boundary detector allows us to
train frame-level and segment-level encoders jointly. Experiments show that our
single model outperforms existing phone and word segmentation methods on TIMIT
and Buckeye datasets. We discover that phone class impacts the boundary
detection performance, and the boundaries between successive vowels or
semivowels are the most difficult to identify. Finally, we use SCPC to extract
speech features at the segment level rather than at uniformly spaced frame
level (e.g., 10 ms) and produce variable rate representations that change
according to the contents of the utterance. We can lower the feature extraction
rate from the typical 100 Hz to as low as 14.5 Hz on average while still
outperforming the MFCC features on the linear phone classification task.

该论文提出了一种使用自监督学习和分段对比预测编码框架来同时完成语音信号的音素和类似单词的分割任务，并发现了连续元音或半元音之间的边界是最难识别的，从而可以在较低的特征提取率下提取出更好的语音特征。

使用分段对比预测编码进行无监督语音分割和可变速率表示学习

Unsupervised Speech Segmentation and Variable Rate Representation  Learning using Segmental Contrastive Predictive Coding

Automatic detection of phoneme or word-like units is one of the core
objectives in zero-resource speech processing. Recent attempts employ
self-supervised training methods, such as contrastive predictive coding (CPC),
where the next frame is predicted given past context. However, CPC only looks
at the audio signal's frame-level structure. We overcome this limitation with a
segmental contrastive predictive coding (SCPC) framework that can model the
signal structure at a higher level e.g. at the phoneme level. In this
framework, a convolutional neural network learns frame-level representation
from the raw waveform via noise-contrastive estimation (NCE). A differentiable
boundary detector finds variable-length segments, which are then used to
optimize a segment encoder via NCE to learn segment representations. The
differentiable boundary detector allows us to train frame-level and
segment-level encoders jointly. Typically, phoneme and word segmentation are
treated as separate tasks. We unify them and experimentally show that our
single model outperforms existing phoneme and word segmentation methods on
TIMIT and Buckeye datasets. We analyze the impact of boundary threshold and
when is the right time to include the segmental loss in the learning process.

本文提出了一个用于零资源语音处理的新的模型，称为分段对比预测编码，能够对音频信号的帧级和更高级别的分组进行编码，并将此模型用于音素和单词分割，结果在 TIMIT 和 Buckeye 数据集上显著优于现有方法。