End-to-end automatic speech recognition (ASR) models aim to learn a generalised speech representation. However, there are limited tools available to understand the internal functions and the effect of hierarchical dependencies within the model architecture. It is crucial to understand the correlations between the layer-wise representations, to derive insights on the relationship between neural representations and performance. Previous investigations of network similarities using correlation analysis techniques have not been explored for End-to-End ASR models. This paper analyses and explores the internal dynamics between layers during training with CNN, LSTM and Transformer based approaches using Canonical correlation analysis (CCA) and centered kernel alignment (CKA) for the experiments. It was found that neural representations within CNN layers exhibit hierarchical correlation dependencies as layer depth increases but this is mostly limited to cases where neural representation correlates more closely. This behaviour is not observed in LSTM architecture, however there is a bottom-up pattern observed across the training process, while Transformer encoder layers exhibit irregular coefficiency correlation as neural depth increases. Altogether, these results provide new insights into the role that neural architectures have upon speech recognition performance. More specifically, these techniques can be used as indicators to build better performing speech recognition models.

本文分析了使用 CNN、LSTM 和 Transformer 等网络模型进行端到端自动语音识别模型训练时层之间的内部动态，通过规范相关性分析和中心核对齐，发现 CNN 层内的神经表示随着层深度的增加而表现出层级相关性依赖性，而这种行为在 LSTM 架构中不会观察到，但在 Transformer 编码器层中会出现不规则的系数相关性随神经深度的增加而增加，这些结果提供了神经体系结构对语音识别性能的作用的新见解，可用作构建更好的语音识别模型的指示器。

端到端语音识别的神经表示洞察