In this paper, we explore an interesting question of what can be obtained
from an $8\times8$ pixel video sequence. Surprisingly, it turns out to be quite
a lot. We show that when we process this $8\times8$ video with the right set of
audio and image priors, we can obtain a full-length, $256\times256$ video. We
achieve this $32\times$ scaling of an extremely low-resolution input using our
novel audio-visual upsampling network. The audio prior helps to recover the
elemental facial details and precise lip shapes and a single high-resolution
target identity image prior provides us with rich appearance details. Our
approach is an end-to-end multi-stage framework. The first stage produces a
coarse intermediate output video that can be then used to animate single target
identity image and generate realistic, accurate and high-quality outputs. Our
approach is simple and performs exceedingly well (an $8\times$ improvement in
FID score) compared to previous super-resolution methods. We also extend our
model to talking-face video compression, and show that we obtain a $3.5\times$
improvement in terms of bits/pixel over the previous state-of-the-art. The
results from our network are thoroughly analyzed through extensive ablation
experiments (in the paper and supplementary material). We also provide the demo
video along with code and models on our website:
http://cvit.iiit.ac.in/research/projects/cvit-projects/talking-face-video-upsampling.

本文探讨了如何从一个 $8	imes8$ 像素的视频序列中获得丰富的信息，通过音频和图像的先验知识以及提出的音视频上采样网络，我们成功实现了 $32	imes$ 的视频放大，进一步在说话人视频压缩领域实现了 $3.5	imes$ 的比特率改进。