Arguably the most common and salient object in daily video communications is
the talking head, as encountered in social media, virtual classrooms,
teleconferences, news broadcasting, talk shows, etc. When communication
bandwidth is limited by network congestions or cost effectiveness, compression
artifacts in talking head videos are inevitable. The resulting video quality
degradation is highly visible and objectionable due to high acuity of human
visual system to faces. To solve this problem, we develop a multi-modality deep
convolutional neural network method for restoring face videos that are
aggressively compressed. The main innovation is a new DCNN architecture that
incorporates known priors of multiple modalities: the video-synchronized speech
signal and semantic elements of the compression code stream, including motion
vectors, code partition map and quantization parameters. These priors strongly
correlate with the latent video and hence they are able to enhance the
capability of deep learning to remove compression artifacts. Ample empirical
evidences are presented to validate the superior performance of the proposed
DCNN method on face videos over the existing state-of-the-art methods.

本文提出了一种多模式深度卷积神经网络方法，用于恢复被激进压缩的脸部视频，并明确了多种模式的先验知识对于减少压缩伪影的重要性，实验证明了该方法对于面部视频的卓越性能。