All previous methods for audio-driven talking head generation assume the
input audio to be clean with a neutral tone. As we show empirically, one can
easily break these systems by simply adding certain background noise to the
utterance or changing its emotional tone (to such as sad). To make talking head
generation robust to such variations, we propose an explicit audio
representation learning framework that disentangles audio sequences into
various factors such as phonetic content, emotional tone, background noise and
others. We conduct experiments to validate that conditioned on disentangled
content representation, the generated mouth movement by our model is
significantly more accurate than previous approaches (without disentangled
learning) in the presence of noise and emotional variations. We further
demonstrate that our framework is compatible with current state-of-the-art
approaches by replacing their original audio learning component with ours. To
our best knowledge, this is the first work which improves the performance of
talking head generation from disentangled audio representation perspective,
which is important for many real-world applications.

提出一种基于声音分解学习的方法来提高音频驱动的视频生成的性能，从而使生成的视频更鲁棒。