This study describes our system for Task 1 Single-speaker Visual Speech
Recognition (VSR) fixed track in the Chinese Continuous Visual Speech
Recognition Challenge (CNVSRC) 2023. Specifically, we use intermediate
connectionist temporal classification (Inter CTC) residual modules to relax the
conditional independence assumption of CTC in our model. Then we use a
bi-transformer decoder to enable the model to capture both past and future
contextual information. In addition, we use Chinese characters as the modeling
units to improve the recognition accuracy of our model. Finally, we use a
recurrent neural network language model (RNNLM) for shallow fusion in the
inference stage. Experiments show that our system achieves a character error
rate (CER) of 38.09% on the Eval set which reaches a relative CER reduction of
21.63% over the official baseline, and obtains a second place in the challenge.

这项研究描述了我们在 2023 年中国持续视觉语音识别挑战赛中针对任务 1 单说话者视觉语音识别（VSR）固定轨道的系统。具体来说，我们在模型中使用中间连接主义时间分类（Inter CTC）残差模块来放宽 CTC 的条件独立性假设，然后使用双变压器解码器使模型能够捕捉过去和未来的上下文信息。此外，我们使用汉字作为建模单元来提高模型的识别准确性。最后，在推断阶段，我们使用了递归神经网络语言模型（RNNLM）进行浅层融合。实验表明，我们的系统在评估集上的字符错误率（CER）为 38.09%，相对于官方基准线减少了 21.63%，并在挑战赛中获得了第二名。

2023 CNVSRC 挑战赛 GUA-Speech 系统介绍

The GUA-Speech System Description for CNVSRC Challenge 2023

Integrating an external language model into a sequence-to-sequence speech
recognition system is non-trivial. Previous works utilize linear interpolation
or a fusion network to integrate external language models. However, these
approaches introduce external components, and increase decoding computation. In
this paper, we instead propose a knowledge distillation based training approach
to integrating external language models into a sequence-to-sequence model. A
recurrent neural network language model, which is trained on large scale
external text, generates soft labels to guide the sequence-to-sequence model
training. Thus, the language model plays the role of the teacher. This approach
does not add any external component to the sequence-to-sequence model during
testing. And this approach is flexible to be combined with shallow fusion
technique together for decoding. The experiments are conducted on public
Chinese datasets AISHELL-1 and CLMAD. Our approach achieves a character error
rate of 9.3%, which is relatively reduced by 18.42% compared with the vanilla
sequence-to-sequence model.

本文提出了一种基于知识蒸馏的训练方法，将外部语言模型集成到序列到序列语音识别系统中，通过递归神经网络语言模型生成软标签来指导序列到序列模型的训练，以此来降低字符错误率。

从语言模型向序列到序列语音识别中迁移拼写知识：向教师学拼写

Learn Spelling from Teachers: Transferring Knowledge from Language  Models to Sequence-to-Sequence Speech Recognition

The standard recurrent neural network language model (RNNLM) generates
sentences one word at a time and does not work from an explicit global sentence
representation. In this work, we introduce and study an RNN-based variational
autoencoder generative model that incorporates distributed latent
representations of entire sentences. This factorization allows it to explicitly
model holistic properties of sentences such as style, topic, and high-level
syntactic features. Samples from the prior over these sentence representations
remarkably produce diverse and well-formed sentences through simple
deterministic decoding. By examining paths through this latent space, we are
able to generate coherent novel sentences that interpolate between known
sentences. We present techniques for solving the difficult learning problem
presented by this model, demonstrate its effectiveness in imputing missing
words, explore many interesting properties of the model's latent sentence
space, and present negative results on the use of the model in language
modeling.

本文介绍一种基于 RNN 和 VAE 的生成模型，该模型可以显式地模拟句子的整体特征，并通过分布式的潜在表示编码句子的风格、主题和高级语法特征。模型生成多样且完整的句子，能够插值生成连贯且新颖的句子，并用于填补缺失的词，同时探索了潜在句子空间的许多有趣属性。