A jump cut offers an abrupt, sometimes unwanted change in the viewing
experience. We present a novel framework for smoothing these jump cuts, in the
context of talking head videos. We leverage the appearance of the subject from
the other source frames in the video, fusing it with a mid-level representation
driven by DensePose keypoints and face landmarks. To achieve motion, we
interpolate the keypoints and landmarks between the end frames around the cut.
We then use an image translation network from the keypoints and source frames,
to synthesize pixels. Because keypoints can contain errors, we propose a
cross-modal attention scheme to select and pick the most appropriate source
amongst multiple options for each key point. By leveraging this mid-level
representation, our method can achieve stronger results than a strong video
interpolation baseline. We demonstrate our method on various jump cuts in the
talking head videos, such as cutting filler words, pauses, and even random
cuts. Our experiments show that we can achieve seamless transitions, even in
the challenging cases where the talking head rotates or moves drastically in
the jump cut.

我们提出了一种新颖的框架来平滑跳切，以说话人视频为背景。我们利用视频中其他源帧中主体的出现来融合密集姿态关键点和面部标记的中级表示。为了实现运动，我们在切点周围的最后帧之间插值关键点和标记。然后，我们使用一个基于关键点和源帧的图像转换网络来合成像素。由于关键点可能存在错误，我们提出了一种跨模态注意机制来选择和选择每个关键点的最合适源选项。通过利用这种中级表示，我们的方法可以获得比强视频插值基准更强的结果。我们在说话人视频的各种跳切情况下展示了我们的方法，例如剪辑过渡词、停顿，甚至随机切换。我们的实验表明，即使在说话人旋转或跳切中发生剧烈运动的挑战性情况下，我们也能实现无缝的过渡。

对话视频中剪辑的平滑化处理

Jump Cut Smoothing for Talking Heads

Virtual Try-On (trying clothes virtually) is a promising application of the
Generative Adversarial Network (GAN). However, it is an arduous task to
transfer the desired clothing item onto the corresponding regions of a human
body because of varying body size, pose, and occlusions like hair and
overlapped clothes. In this paper, we try to produce photo-realistic translated
images through semantic segmentation and a generative adversarial
architecture-based image translation network. We present a novel image-based
Virtual Try-On application VTON-IT that takes an RGB image, segments desired
body part, and overlays target cloth over the segmented body region. Most
state-of-the-art GAN-based Virtual Try-On applications produce unaligned
pixelated synthesis images on real-life test images. However, our approach
generates high-resolution natural images with detailed textures on such variant
images.

使用基于生成对抗网络的图像翻译网络和语义分割技术，通过 VTON-IT 图像虚拟试穿应用程序将目标服装覆盖在感兴趣的身体部位，并生成细节丰富的高分辨率自然图像。

VTON-IT: 图像翻译的虚拟试衣

VTON-IT: Virtual Try-On using Image Translation

Current methods for single-image depth estimation use training datasets with
real image-depth pairs or stereo pairs, which are not easy to acquire. We
propose a framework, trained on synthetic image-depth pairs and unpaired real
images, that comprises an image translation network for enhancing realism of
input images, followed by a depth prediction network. A key idea is having the
first network act as a wide-spectrum input translator, taking in either
synthetic or real images, and ideally producing minimally modified realistic
images. This is done via a reconstruction loss when the training input is real,
and GAN loss when synthetic, removing the need for heuristic
self-regularization. The second network is trained on a task loss for synthetic
image-depth pairs, with extra GAN loss to unify real and synthetic feature
distributions. Importantly, the framework can be trained end-to-end, leading to
good results, even surpassing early deep-learning methods that use real paired
data.

本文提出一种基于图像翻译网络和深度预测网络的框架，使用合成的图像 - 深度对和非配对的真实图像进行训练，实现单图深度估计。该框架可以端到端训练，得到好的结果，甚至超过使用真实配对数据的早期深度学习方法。