Transformers have become a key architecture in speech processing, but our
understanding of how they build up representations of acoustic and linguistic
structure is limited. In this study, we address this gap by investigating how
measures of 'context-mixing' developed for text models can be adapted and
applied to models of spoken language. We identify a linguistic phenomenon that
is ideal for such a case study: homophony in French (e.g. livre vs livres),
where a speech recognition model has to attend to syntactic cues such as
determiners and pronouns in order to disambiguate spoken words with identical
pronunciations and transcribe them while respecting grammatical agreement. We
perform a series of controlled experiments and probing analyses on
Transformer-based speech models. Our findings reveal that representations in
encoder-only models effectively incorporate these cues to identify the correct
transcription, whereas encoders in encoder-decoder models mainly relegate the
task of capturing contextual dependencies to decoder modules.

通过研究如何将文本模型的 ' 上下文混合 ' 适用于口语语言模型，揭示了 Transformer 模型如何通过关注句法线索来解决同音词消歧问题，并发现编码器与解码器模块在捕捉上下文依赖性任务中的不同表现。