Lipreading is the task of decoding text from the movement of a speaker's
mouth. Traditional approaches separated the problem into two stages: designing
or learning visual features, and prediction. More recent deep lipreading
approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman,
2016a). However, existing work on models trained end-to-end perform only word
classification, rather than sentence-level sequence prediction. Studies have
shown that human lipreading performance increases for longer words (Easton &
Basala, 1982), indicating the importance of features capturing temporal context
in an ambiguous communication channel. Motivated by this observation, we
present LipNet, a model that maps a variable-length sequence of video frames to
text, making use of spatiotemporal convolutions, a recurrent network, and the
connectionist temporal classification loss, trained entirely end-to-end. To the
best of our knowledge, LipNet is the first end-to-end sentence-level lipreading
model that simultaneously learns spatiotemporal visual features and a sequence
model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level,
overlapped speaker split task, outperforming experienced human lipreaders and
the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).

LipNet 是一个全新的、端到端的实时语音转文字模型，它能够将连续的视频帧序列准确地映射成文本，并能够同时学习时空视觉特征和序列模型，它在句子级别的听力阅读任务上表现出色，实现了 95.2％的精度，优于先前研究的单词级准确度。