In the field of portrait video generation, the use of single images to
generate portrait videos has become increasingly prevalent. A common approach
involves leveraging generative models to enhance adapters for controlled
generation. However, control signals (e.g., text, audio, reference image, pose,
depth map, etc.) can vary in strength. Among these, weaker conditions often
struggle to be effective due to interference from stronger conditions, posing a
challenge in balancing these conditions. In our work on portrait video
generation, we identified audio signals as particularly weak, often
overshadowed by stronger signals such as facial pose and reference image.
However, direct training with weak signals often leads to difficulties in
convergence. To address this, we propose V-Express, a simple method that
balances different control signals through the progressive training and the
conditional dropout operation. Our method gradually enables effective control
by weak conditions, thereby achieving generation capabilities that
simultaneously take into account the facial pose, reference image, and audio.
The experimental results demonstrate that our method can effectively generate
portrait videos controlled by audio. Furthermore, a potential solution is
provided for the simultaneous and effective use of conditions of varying
strengths.

通过渐进式训练和条件丢弃操作，我们提出了一种名为 V-Express 的简单方法，平衡了不同的控制信号，使得我们的系统能够同时考虑面部姿势、参考图像和音频，并在实验证明我们的方法能够有效地生成受音频控制的肖像视频，为不同强度的条件的同时和有效使用提供了潜在解决方案。