Audio-visual speech recognition (AVSR) has gained remarkable success for
ameliorating the noise-robustness of speech recognition. Mainstream methods
focus on fusing audio and visual inputs to obtain modality-invariant
representations. However, such representations are prone to over-reliance on
audio modality as it is much easier to recognize than video modality in clean
conditions. As a result, the AVSR model underestimates the importance of visual
stream in face of noise corruption. To this end, we leverage visual
modality-specific representations to provide stable complementary information
for the AVSR task. Specifically, we propose a reinforcement learning (RL) based
framework called MSRL, where the agent dynamically harmonizes
modality-invariant and modality-specific representations in the auto-regressive
decoding process. We customize a reward function directly related to
task-specific metrics (i.e., word error rate), which encourages the MSRL to
effectively explore the optimal integration strategy. Experimental results on
the LRS3 dataset show that the proposed method achieves state-of-the-art in
both clean and various noisy conditions. Furthermore, we demonstrate the better
generality of MSRL system than other baselines when test set contains unseen
noises.

我们提出了一个基于强化学习的框架（MSRL），动态地协调模态不变和模态特异性的表示，从而稳定地提供互补信息，用于音视频语音识别任务，实验结果表明，此方法在 LRS3 数据集中取得了最新的成果。