A jump cut offers an abrupt, sometimes unwanted change in the viewing
experience. We present a novel framework for smoothing these jump cuts, in the
context of talking head videos. We leverage the appearance of the subject from
the other source frames in the video, fusing it with a mid-level representation
driven by DensePose keypoints and face landmarks. To achieve motion, we
interpolate the keypoints and landmarks between the end frames around the cut.
We then use an image translation network from the keypoints and source frames,
to synthesize pixels. Because keypoints can contain errors, we propose a
cross-modal attention scheme to select and pick the most appropriate source
amongst multiple options for each key point. By leveraging this mid-level
representation, our method can achieve stronger results than a strong video
interpolation baseline. We demonstrate our method on various jump cuts in the
talking head videos, such as cutting filler words, pauses, and even random
cuts. Our experiments show that we can achieve seamless transitions, even in
the challenging cases where the talking head rotates or moves drastically in
the jump cut.

我们提出了一种新颖的框架来平滑跳切，以说话人视频为背景。我们利用视频中其他源帧中主体的出现来融合密集姿态关键点和面部标记的中级表示。为了实现运动，我们在切点周围的最后帧之间插值关键点和标记。然后，我们使用一个基于关键点和源帧的图像转换网络来合成像素。由于关键点可能存在错误，我们提出了一种跨模态注意机制来选择和选择每个关键点的最合适源选项。通过利用这种中级表示，我们的方法可以获得比强视频插值基准更强的结果。我们在说话人视频的各种跳切情况下展示了我们的方法，例如剪辑过渡词、停顿，甚至随机切换。我们的实验表明，即使在说话人旋转或跳切中发生剧烈运动的挑战性情况下，我们也能实现无缝的过渡。