Attention-based encoder-decoder (AED) models have shown impressive
performance in ASR. However, most existing AED methods neglect to
simultaneously leverage both acoustic and semantic features in decoder, which
is crucial for generating more accurate and informative semantic states. In
this paper, we propose an Acoustic and Semantic Cooperative Decoder (ASCD) for
ASR. In particular, unlike vanilla decoders that process acoustic and semantic
features in two separate stages, ASCD integrates them cooperatively. To prevent
information leakage during training, we design a Causal Multimodal Mask.
Moreover, a variant Semi-ASCD is proposed to balance accuracy and computational
cost. Our proposal is evaluated on the publicly available AISHELL-1 and
aidatatang_200zh datasets using Transformer, Conformer, and Branchformer as
encoders, respectively. The experimental results show that ASCD significantly
improves the performance by leveraging both the acoustic and semantic
information cooperatively.

该论文提出了一种声学和语义协同解码器 ASRD，可以同时利用声学和语义特征来提高自动语音识别的性能。通过引入因果多模态屏蔽，可以防止在训练期间的信息泄漏，并且还提出了一种改良版的半监督 ASCD 来平衡准确性和计算成本。实验结果表明，ASRD 可以显著提高 ASR 的准确性。