Automatic pronunciation assessment is a major component of a
computer-assisted pronunciation training system. To provide in-depth feedback,
scoring pronunciation at various levels of granularity such as phoneme, word,
and utterance, with diverse aspects such as accuracy, fluency, and
completeness, is essential. However, existing multi-aspect multi-granularity
methods simultaneously predict all aspects at all granularity levels;
therefore, they have difficulty in capturing the linguistic hierarchy of
phoneme, word, and utterance. This limitation further leads to neglecting
intimate cross-aspect relations at the same linguistic unit. In this paper, we
propose a Hierarchical Pronunciation Assessment with Multi-aspect Attention
(HiPAMA) model, which hierarchically represents the granularity levels to
directly capture their linguistic structures and introduces multi-aspect
attention that reflects associations across aspects at the same level to create
more connotative representations. By obtaining relational information from both
the granularity- and aspect-side, HiPAMA can take full advantage of multi-task
learning. Remarkable improvements in the experimental results on the
speachocean762 datasets demonstrate the robustness of HiPAMA, particularly in
the difficult-to-assess aspects.

本研究提出了一种基于多方面注意力和分层表示的自动发音评估模型（ HiPAMA ），可在不同层次（如音素、单词和句子）上评估发音的多个方面（ 如准确性、流利度和完整性），从而实现更全面的反馈评估

多方面关注的层次发音评估

Hierarchical Pronunciation Assessment with Multi-Aspect Attention

Mispronunciation detection and diagnosis (MDD) technology is a key component
of computer-assisted pronunciation training system (CAPT). In the field of
assessing the pronunciation quality of constrained speech, the given
transcriptions can play the role of a teacher. Conventional methods have fully
utilized the prior texts for the model construction or improving the system
performance, e.g. forced-alignment and extended recognition networks. Recently,
some end-to-end based methods attempt to incorporate the prior texts into model
training and preliminarily show the effectiveness. However, previous studies
mostly consider applying raw attention mechanism to fuse audio representations
with text representations, without taking possible text-pronunciation mismatch
into account. In this paper, we present a gating strategy that assigns more
importance to the relevant audio features while suppressing irrelevant text
information. Moreover, given the transcriptions, we design an extra contrastive
loss to reduce the gap between the learning objective of phoneme recognition
and MDD. We conducted experiments using two publicly available datasets (TIMIT
and L2-Arctic) and our best model improved the F1 score from $57.51\%$ to
$61.75\%$ compared to the baselines. Besides, we provide a detailed analysis to
shed light on the effectiveness of gating mechanism and contrastive learning on
MDD.

本文设计了门控策略以及对比学习等方法，以便更好地利用先前给定的文本信息作为指导，有效地提高了自动语音纠音技术的性能。