Recognizing speech from silent lip movement, which is called lip reading, is
a challenging task due to 1) the inherent information insufficiency of lip
movement to fully represent the speech, and 2) the existence of homophenes that
have similar lip movement with different pronunciations. In this paper, we try
to alleviate the aforementioned two challenges in lip reading by proposing a
Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with audio-visual
datasets and remembers audio representations by modelling the
inter-relationships of paired audio-visual representations. At the inference
stage, visual input alone can extract the saved audio representation from the
memory by examining the learned inter-relationships. Therefore, the lip reading
model can complement the insufficient visual information with the extracted
audio representations. Secondly, MVM is composed of multi-head key memories for
saving visual features and one value memory for saving audio knowledge, which
is designed to distinguish the homophenes. With the multi-head key memories,
MVM extracts possible candidate audio features from the memory, which allows
the lip reading model to consider the possibility of which pronunciations can
be represented from the input lip movement. This also can be viewed as an
explicit implementation of the one-to-many mapping of viseme-to-phoneme.
Moreover, MVM is employed in multi-temporal levels to consider the context when
retrieving the memory and distinguish the homophenes. Extensive experimental
results verify the effectiveness of the proposed method in lip reading and in
distinguishing the homophenes.

本文提出一种多头视听记忆模型 (MVM)，用于减轻唇读中信息不足和同音异义现象的挑战。该模型由音频 - 视觉数据集训练得到，结合多头记忆键和值来保存可区分同音异义现象的音频和视觉特征，并且在考虑上下文语境的多时间级别下使用，有效提高了唇读的识别率和准确性。