The goal of this paper is to develop state-of-the-art models for lip reading
-- visual speech recognition. We develop three architectures and compare their
accuracy and training times: (i) a recurrent model using LSTMs; (ii) a fully
convolutional model; and (iii) the recently proposed transformer model. The
recurrent and fully convolutional models are trained with a Connectionist
Temporal Classification loss and use an explicit language model for decoding,
the transformer is a sequence-to-sequence model. Our best performing model
improves the state-of-the-art word error rate on the challenging BBC-Oxford Lip
Reading Sentences 2 (LRS2) benchmark dataset by over 20 percent.
As a further contribution we investigate the fully convolutional model when
used for online (real time) lip reading of continuous speech, and show that it
achieves high performance with low latency.

本文旨在发展最先进的口型阅读模型，分别使用 LSTM 递归模型、全卷积模型和最近提出的 Transformer 模型，并通过 BBC-Oxford Lip Reading Sentences 2 (LRS2) 基准数据集的实验表明了本研究的最佳成果。

深度唇语识别：模型比较和在线应用

Deep Lip Reading: a comparison of models and an online application

In this study, we propose a deep neural network for reconstructing
intelligible speech from silent lip movement videos. We use auditory
spectrogram as spectral representation of speech and its corresponding sound
generation method resulting in a more natural sounding reconstructed speech.
Our proposed network consists of an autoencoder to extract bottleneck features
from the auditory spectrogram which is then used as target to our main lip
reading network comprising of CNN, LSTM and fully connected layers. Our
experiments show that the autoencoder is able to reconstruct the original
auditory spectrogram with a 98% correlation and also improves the quality of
reconstructed speech from the main lip reading network. Our model, trained
jointly on different speakers is able to extract individual speaker
characteristics and gives promising results of reconstructing intelligible
speech with superior word recognition accuracy.

通过一个由自动编码器和卷积神经网络构成的模型，在无声视频中对语音信号的频谱表示进行重建，获得了 98% 的相关性并且提高了音频质量。经过多个说话人的联合训练，该模型能够提取出不同说话人的个体特征，具有良好的识别准确率。

Lip2AudSpec: 从静态唇部运动视频重建语音

Lip2AudSpec: Speech reconstruction from silent lip movements video

The goal of this work is to recognise phrases and sentences being spoken by a
talking face, with or without the audio. Unlike previous works that have
focussed on recognising a limited number of words or phrases, we tackle lip
reading as an open-world problem - unconstrained natural language sentences,
and in the wild videos.
Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WLAS)
network that learns to transcribe videos of mouth motion to characters; (2) a
curriculum learning strategy to accelerate training and to reduce overfitting;
(3) a 'Lip Reading Sentences' (LRS) dataset for visual speech recognition,
consisting of over 100,000 natural sentences from British television.
The WLAS model trained on the LRS dataset surpasses the performance of all
previous work on standard lip reading benchmark datasets, often by a
significant margin. This lip reading performance beats a professional lip
reader on videos from BBC television, and we also demonstrate that visual
information helps to improve speech recognition performance even when the audio
is available.

本研究目的在于通过观察说话者的口型运动，并结合音频或不结合音频，识别出单词词组和句子。我们提出了一种 “看、听、关注和拼写”（WLAS）网络来学习将口腔运动视频转录为字符的方法，设计了一种课程学习策略来加速训练并减少过拟合，建立了一个视觉语音识别数据集 - “Lip Reading Sentences”（LRS）数据集，其中包含来自英国电视的超过 100,000 个自然语句，该方法优于以往所有标准嘴唇读取基准数据集的表现。