Talking face generation has been extensively investigated owing to its wide
applicability. The two primary frameworks used for talking face generation
comprise a text-driven framework, which generates synchronized speech and
talking faces from text, and a speech-driven framework, which generates talking
faces from speech. To integrate these frameworks, this paper proposes a unified
facial landmark generator (UniFLG). The proposed system exploits end-to-end
text-to-speech not only for synthesizing speech but also for extracting a
series of latent representations that are common to text and speech, and feeds
it to a landmark decoder to generate facial landmarks. We demonstrate that our
system achieves higher naturalness in both speech synthesis and facial landmark
generation compared to the state-of-the-art text-driven method. We further
demonstrate that our system can generate facial landmarks from speech of
speakers without facial video data or even speech data.

提出一种统一的面部标记生成器（Unified facial landmark generator），利用端到端的文本到语音不仅用于合成语音，还用于提取一系列与文本和语音共同的潜在表示，将其馈送至标记解码器以生成面部标记，并演示这个系统在语音合成和面部地标生成方面比最先进的基于文本的方法具有更高的真实性，可以从没有面部视频数据甚至语音数据的演讲者的语音中生成面部特征点。

文本或语音驱动的统一人脸标记生成器

UniFLG: Unified Facial Landmark Generator from Text or Speech

High-quality HDRIs(High Dynamic Range Images), typically HDR panoramas, are
one of the most popular ways to create photorealistic lighting and 360-degree
reflections of 3D scenes in graphics. Given the difficulty of capturing HDRIs,
a versatile and controllable generative model is highly desired, where layman
users can intuitively control the generation process. However, existing
state-of-the-art methods still struggle to synthesize high-quality panoramas
for complex scenes. In this work, we propose a zero-shot text-driven framework,
Text2Light, to generate 4K+ resolution HDRIs without paired training data.
Given a free-form text as the description of the scene, we synthesize the
corresponding HDRI with two dedicated steps: 1) text-driven panorama generation
in low dynamic range(LDR) and low resolution, and 2) super-resolution inverse
tone mapping to scale up the LDR panorama both in resolution and dynamic range.
Specifically, to achieve zero-shot text-driven panorama generation, we first
build dual codebooks as the discrete representation for diverse environmental
textures. Then, driven by the pre-trained CLIP model, a text-conditioned global
sampler learns to sample holistic semantics from the global codebook according
to the input text. Furthermore, a structure-aware local sampler learns to
synthesize LDR panoramas patch-by-patch, guided by holistic semantics. To
achieve super-resolution inverse tone mapping, we derive a continuous
representation of 360-degree imaging from the LDR panorama as a set of
structured latent codes anchored to the sphere. This continuous representation
enables a versatile module to upscale the resolution and dynamic range
simultaneously. Extensive experiments demonstrate the superior capability of
Text2Light in generating high-quality HDR panoramas. In addition, we show the
feasibility of our work in realistic rendering and immersive VR.

本文中提出了一种零样本文本驱动框架 (Text2Light)，它使用自然语言对场景进行描述，并生成适用于复杂场景的 4K + 分辨率 HDR 全景图，其中使用了双重代码本来生成各种环境纹理的离散表示，并由经过预训练的 CLIP 模型驱动的全局采样器和结构感知本地采样器进行指导。同时，提出了一种 360 度图像的连续表示方法，以支持同时提高分辨率和动态范围。