It has been generally assumed in the automatic speech recognition (ASR) literature that it is better for models to have access to wider context windows. Yet, many of the potential reasons this might be true in the supervised setting do not necessarily transfer over to the case of unsupervised learning. We investigate how much context is necessary to achieve high-quality pre-trained acoustic models using self-supervised learning. We principally investigate contrastive predictive coding (CPC), which we adapt to be able to precisely control the amount of context visible to the model during training and inference. We find that phone discriminability in the resulting model representations peaks at around 40~ms of preceding context, and that having too much context (beyond around 320 ms) substantially degrades the quality of the representations. Surprisingly, we find that this pattern also transfers to supervised ASR when the pre-trained representations are used as frozen input features. Our results point to potential changes in the design of current upstream architectures to better facilitate a variety of downstream tasks.

调查了自监督学习中使用多少上下文能够实现高质量的预训练声学模型，发现在训练和推理过程中，40ms的上下文能够达到最佳的音素可辨识性，而太多上下文则会显著降低表示质量，并且这种模式在监督ASR中也适用，当预训练表示被用作冻结输入特征时。研究结果指出了当前上游架构设计上可能需要进行的变化以更好地支持各种下游任务。

上下文大小对语音预训练的影响：更大并不总是更好