Machine lipreading is a special type of automatic speech recognition (ASR)
which transcribes human speech by visually interpreting the movement of related
face regions including lips, face, and tongue. Recently, deep neural network
based lipreading methods show great potential and have exceeded the accuracy of
experienced human lipreaders in some benchmark datasets. However, lipreading is
still far from being solved, and existing methods tend to have high error rates
on the wild data. In this paper, we propose LCANet, an end-to-end deep neural
network based lipreading system. LCANet encodes input video frames using a
stacked 3D convolutional neural network (CNN), highway network and
bidirectional GRU network. The encoder effectively captures both short-term and
long-term spatio-temporal information. More importantly, LCANet incorporates a
cascaded attention-CTC decoder to generate output texts. By cascading CTC with
attention, it partially eliminates the defect of the conditional independence
assumption of CTC within the hidden neural layers, and this yields notably
performance improvement as well as faster convergence. The experimental results
show the proposed system achieves a 1.3% CER and 3.0% WER on the GRID corpus
database, leading to a 12.3% improvement compared to the state-of-the-art
methods.

本文提出了一种基于端到端深度神经网络的 LCANet 口型识别系统，利用 3D CNN、highway network 和双向 GRU network 来捕捉短期和长期的空时信息，并采用连续的注意力 - CTC 解码器来生成输出文字，实验结果表明，与现有的最先进方法相比，所提出的系统在 GRID 语料库上取得了 1.3% 的 CER 和 3.0% 的 WER，有 12.3% 的改进。

LCANet: 级联注意力 - CTC 的端到端唇语识别

LCANet: End-to-End Lipreading with Cascaded Attention-CTC

Machine lipreading (MLR) is speech recognition from visual cues and a niche
research problem in speech processing & computer vision. Current challenges
fall into two groups: the content of the video, such as rate of speech or; the
parameters of the video recording e.g, video resolution. We show that HD video
is not needed to successfully lipread with a computer. The term "viseme" is
used in machine lipreading to represent a visual cue or gesture which
corresponds to a subgroup of phonemes where the phonemes are visually
indistinguishable. A phoneme is the smallest sound one can utter, because there
are more phonemes per viseme, maps between units show a many-to-one
relationship. Many maps have been presented, we compare these and our results
show Lee's is best. We propose a new method of speaker-dependent
phoneme-to-viseme maps and compare these to Lee's. Our results show the
sensitivity of phoneme clustering and we use our new knowledge to augment a
conventional MLR system. It has been observed in MLR, that classifiers need
training on test subjects to achieve accuracy. Thus machine lipreading is
highly speaker-dependent. Conversely speaker independence is robust
classification of non-training speakers. We investigate the dependence of
phoneme-to-viseme maps between speakers and show there is not a high
variability of visemes, but there is high variability in trajectory between
visemes of individual speakers with the same ground truth. This implies a
dependency upon the number of visemes within each set for each individual. We
show that prior phoneme-to-viseme maps rarely have enough visemes and the
optimal size, which varies by speaker, ranges from 11-35. Finally we decode
from visemes back to phonemes and into words. Our novel approach uses the
optimum range visemes within hierarchical training of phoneme classifiers and
demonstrates a significant increase in classification accuracy.

该研究提出了一种新的说话者依赖性的音素 - 面部表情地图方法，以及从面部表情解码到音素和单词的新方法，展示了在训练音素分类器的最佳范围面部表情中的层次化训练可以显著提高分类准确性。