The human perception system is often assumed to recruit motor knowledge when
processing auditory speech inputs. Using articulatory modeling and deep
learning, this study examines how this articulatory information can be used for
discovering speech units in a self-supervised setting. We used vector-quantized
variational autoencoders (VQ-VAE) to learn discrete representations from
articulatory and acoustic speech data. In line with the zero-resource paradigm,
an ABX test was then used to investigate how the extracted representations
encode phonetically relevant properties. Experiments were conducted on three
different corpora in English and French. We found that articulatory information
rather organises the latent representations in terms of place of articulation
whereas the speech acoustics mainly structure the latent space in terms of
manner of articulation. We show that an optimal fusion of the two modalities
can lead to a joint representation of these phonetic dimensions more accurate
than each modality considered individually. Since articulatory information is
usually not available in a practical situation, we finally investigate the
benefit it provides when inferred from the speech acoustics in a
self-supervised manner.

使用关节建模和深度学习研究了如何在自我监督情况下使用关节信息发现语音单元，并在实验中发现联合使用关节和语音学模态可以更准确地表示这些语音学维度。