Visual Speech Recognition (VSR) is the process of recognizing or interpreting
speech by watching the lip movements of the speaker. Recent machine learning
based approaches model VSR as a classification problem; however, the scarcity
of training data leads to error-prone systems with very low accuracies in
predicting unseen classes. To solve this problem, we present a novel approach
to zero-shot learning by generating new classes using Generative Adversarial
Networks (GANs), and show how the addition of unseen class samples increases
the accuracy of a VSR system by a significant margin of 27% and allows it to
handle speaker-independent out-of-vocabulary phrases. We also show that our
models are language agnostic and therefore capable of seamlessly generating,
using English training data, videos for a new language (Hindi). To the best of
our knowledge, this is the first work to show empirical evidence of the use of
GANs for generating training samples of unseen classes in the domain of VSR,
hence facilitating zero-shot learning. We make the added videos for new classes
publicly available along with our code.

本文介绍了一种使用生成对抗网络 (GANs) 生成新类别的零样本学习方法，用于提高视觉语音识别系统的准确性和处理跨语音的词汇。本研究展示出，该模型是语言无关的，因此可以使用英语训练数据生成新语言 (如印地语) 的视频。我们公开了添加了新类别的视频和代码。