Visual Speech Recognition (VSR) is a task to predict a sentence or word from
lip movements. Some works have been recently presented which use audio signals
to supplement visual information. However, existing methods utilize only
limited information such as phoneme-level features and soft labels of Automatic
Speech Recognition (ASR) networks. In this paper, we present a Multi-Temporal
Lip-Audio Memory (MTLAM) that makes the best use of audio signals to complement
insufficient information of lip movements. The proposed method is mainly
composed of two parts: 1) MTLAM saves multi-temporal audio features produced
from short- and long-term audio signals, and the MTLAM memorizes a
visual-to-audio mapping to load stored multi-temporal audio features from
visual features at the inference phase. 2) We design an audio temporal model to
produce multi-temporal audio features capturing the context of neighboring
words. In addition, to construct effective visual-to-audio mapping, the audio
temporal models can generate audio features time-aligned with visual features.
Through extensive experiments, we validate the effectiveness of the MTLAM
achieving state-of-the-art performances on two public VSR datasets.

本文提出了一种利用多时间点音频记忆的视觉语音识别方法，将音频信号与唇部运动相结合，实践证明此方法在两个公共视觉语音识别数据集上取得了最新技术表现。