Automatic lip-reading (ALR) aims to automatically transcribe spoken content
from a speaker's silent lip motion captured in video. Current mainstream
lip-reading approaches only use a single visual encoder to model input videos
of a single scale. In this paper, we propose to enhance lipreading by
incorporating multi-scale video data and multi-encoder. Specifically, we first
propose a novel multi-scale lip extraction algorithm based on the size of the
speaker's face and an enhanced ResNet3D visual front-end (VFE) to extract lip
features at different scales. For the multi-encoder, in addition to the
mainstream Transformer and Conformer, we also incorporate the recently proposed
Branchformer and EBranchformer as visual encoders. In the experiments, we
explore the influence of different video data scales and encoders on ALR system
performance and fuse the texts transcribed by all ALR systems using recognizer
output voting error reduction (ROVER). Finally, our proposed approach placed
second in the ICME 2024 ChatCLR Challenge Task 2, with a 21.52% reduction in
character error rate (CER) compared to the official baseline on the evaluation
set.

提出了一种改善自动唇读技术的方法，通过多尺度视频数据和多编码器来提取不同尺度的唇部特征，并在 ICME 2024 ChatCLR 挑战中取得了第二名的成绩，与官方基准相比字符错误率减少了 21.52%。