Recent advancements in human video synthesis have enabled the generation of
high-quality videos through the application of stable diffusion models.
However, existing methods predominantly concentrate on animating solely the
human element (the foreground) guided by pose information, while leaving the
background entirely static. Contrary to this, in authentic, high-quality
videos, backgrounds often dynamically adjust in harmony with foreground
movements, eschewing stagnancy. We introduce a technique that concurrently
learns both foreground and background dynamics by segregating their movements
using distinct motion representations. Human figures are animated leveraging
pose-based motion, capturing intricate actions. Conversely, for backgrounds, we
employ sparse tracking points to model motion, thereby reflecting the natural
interaction between foreground activity and environmental changes. Training on
real-world videos enhanced with this innovative motion depiction approach, our
model generates videos exhibiting coherent movement in both foreground subjects
and their surrounding contexts. To further extend video generation to longer
sequences without accumulating errors, we adopt a clip-by-clip generation
strategy, introducing global features at each step. To ensure seamless
continuity across these segments, we ingeniously link the final frame of a
produced clip with input noise to spawn the succeeding one, maintaining
narrative flow. Throughout the sequential generation process, we infuse the
feature representation of the initial reference image into the network,
effectively curtailing any cumulative color inconsistencies that may otherwise
arise. Empirical evaluations attest to the superiority of our method in
producing videos that exhibit harmonious interplay between foreground actions
and responsive background dynamics, surpassing prior methodologies in this
regard.

通过采用不同的运动表示方法，我们引入了一种可同时学习前景和背景动力学的技术，该技术通过把它们的运动分离开来。我们的模型在真实世界的视频训练中生成了前景和背景动作之间协调的视频，并通过引入全局特征来扩展视频生成到更长的序列，以确保平滑的连续性。这种方法在产生展示前景动作和反应背景动态的视频方面优于之前的方法。

人体视频生成中增强逼真度的前景与背景动作解耦

Disentangling Foreground and Background Motion for Enhanced Realism in  Human Video Generation

Spatio-temporal coherency is a major challenge in synthesizing high quality
videos, particularly in synthesizing human videos that contain rich global and
local deformations. To resolve this challenge, previous approaches have
resorted to different features in the generation process aimed at representing
appearance and motion. However, in the absence of strict mechanisms to
guarantee such disentanglement, a separation of motion from appearance has
remained challenging, resulting in spatial distortions and temporal jittering
that break the spatio-temporal coherency. Motivated by this, we here propose
LEO, a novel framework for human video synthesis, placing emphasis on
spatio-temporal coherency. Our key idea is to represent motion as a sequence of
flow maps in the generation process, which inherently isolate motion from
appearance. We implement this idea via a flow-based image animator and a Latent
Motion Diffusion Model (LMDM). The former bridges a space of motion codes with
the space of flow maps, and synthesizes video frames in a warp-and-inpaint
manner. LMDM learns to capture motion prior in the training data by
synthesizing sequences of motion codes. Extensive quantitative and qualitative
analysis suggests that LEO significantly improves coherent synthesis of human
videos over previous methods on the datasets TaichiHD, FaceForensics and
CelebV-HQ. In addition, the effective disentanglement of appearance and motion
in LEO allows for two additional tasks, namely infinite-length human video
synthesis, as well as content-preserving video editing.

提出了一种名为 LEO 的新颖方法，通过将运动表示为一系列流场图，在生成过程中固有地分离运动和外貌，从而强调时空一致性，并取得了比以前方法更好的人类视频合成效果。

LEO：人体视频合成的生成式潜空间图像动画器

LEO: Generative Latent Image Animator for Human Video Synthesis

Synthesizing realistic videos of humans using neural networks has been a
popular alternative to the conventional graphics-based rendering pipeline due
to its high efficiency. Existing works typically formulate this as an
image-to-image translation problem in 2D screen space, which leads to artifacts
such as over-smoothing, missing body parts, and temporal instability of
fine-scale detail, such as pose-dependent wrinkles in the clothing. In this
paper, we propose a novel human video synthesis method that approaches these
limiting factors by explicitly disentangling the learning of time-coherent
fine-scale details from the embedding of the human in 2D screen space. More
specifically, our method relies on the combination of two convolutional neural
networks (CNNs). Given the pose information, the first CNN predicts a dynamic
texture map that contains time-coherent high-frequency details, and the second
CNN conditions the generation of the final video on the temporally coherent
output of the first CNN. We demonstrate several applications of our approach,
such as human reenactment and novel view synthesis from monocular video, where
we show significant improvement over the state of the art both qualitatively
and quantitatively.

本文介绍了一种新的人类视频合成方法，通过两个卷积神经网络（CNNs）明确分离 2D 屏幕空间中的人类嵌入和时间相干的细节，其中第一个 CNN 预测一个动态纹理映射，第二个 CNN 条件下第一个 CNN 的输出生成最终的视频。 该方法被应用于人类重演和单目视频的新视图合成等多个领域，并且在质量和数量方面都有显着的提高。