This paper delineates the visual speech recognition (VSR) system introduced
by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech
Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of
Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task. In terms
of data processing, we leverage the lip motion extractor from the baseline1 to
produce multi-scale video data. Besides, various augmentation techniques are
applied during training, encompassing speed perturbation, random rotation,
horizontal flipping, and color transformation. The VSR model adopts an
end-to-end architecture with joint CTC/attention loss, comprising a ResNet3D
visual frontend, an E-Branchformer encoder, and a Transformer decoder.
Experiments show that our system achieves 34.76% CER for the Single-Speaker
Task and 41.06% CER for the Multi-Speaker Task after multi-system fusion,
ranking first place in all three tracks we participate.

本文介绍了 NPU-ASLP-LiAuto（第 237 队）在 2023 年中国连续视觉语音识别挑战赛中引入的视觉语音识别（VSR）系统，涉及单说话人 VSR 任务的固定和开放跟踪，以及多说话人 VSR 任务的开放跟踪。实验结果表明，在多系统融合后，我们的系统在单说话人任务上实现了 34.76% 的 CER，在多说话人任务上实现了 41.06% 的 CER，在我们参与的所有三个跟踪中排名第一。