Lip reading aims to predict speech based on lip movements alone. As it
focuses on visual information to model the speech, its performance is
inherently sensitive to personal lip appearances and movements. This makes the
lip reading models show degraded performance when they are applied to unseen
speakers due to the mismatch between training and testing conditions. Speaker
adaptation technique aims to reduce this mismatch between train and test
speakers, thus guiding a trained model to focus on modeling the speech content
without being intervened by the speaker variations. In contrast to the efforts
made in audio-based speech recognition for decades, the speaker adaptation
methods have not well been studied in lip reading. In this paper, to remedy the
performance degradation of lip reading model on unseen speakers, we propose a
speaker-adaptive lip reading method, namely user-dependent padding. The
user-dependent padding is a speaker-specific input that can participate in the
visual feature extraction stage of a pre-trained lip reading model. Therefore,
the lip appearances and movements information of different speakers can be
considered during the visual feature encoding, adaptively for individual
speakers. Moreover, the proposed method does not need 1) any additional layers,
2) to modify the learned weights of the pre-trained model, and 3) the speaker
label of train data used during pre-train. It can directly adapt to unseen
speakers by learning the user-dependent padding only, in a supervised or
unsupervised manner. Finally, to alleviate the speaker information
insufficiency in public lip reading databases, we label the speaker of a
well-known audio-visual database, LRW, and design an unseen-speaker lip reading
scenario named LRW-ID.

本文提出了一种音频读唇的说话人自适应方法，该方法针对模型训练与测试时说话人不匹配的情况，使用特定的输入（称为用户相关填充）参与预先训练的模型的视觉特征提取阶段，以进行适应性的个人化的视觉特征编码。