Generating face image with specific gaze information has attracted
considerable attention. Existing approaches typically input gaze values
directly for face generation, which is unnatural and requires annotated gaze
datasets for training, thereby limiting its application. In this paper, we
present a novel gaze-controllable face generation task. Our approach inputs
textual descriptions that describe human gaze and head behavior and generates
corresponding face images. Our work first introduces a text-of-gaze dataset
containing over 90k text descriptions spanning a dense distribution of gaze and
head poses. We further propose a gaze-controllable text-to-face method. Our
method contains a sketch-conditioned face diffusion module and a model-based
sketch diffusion module. We define a face sketch based on facial landmarks and
eye segmentation map. The face diffusion module generates face images from the
face sketch, and the sketch diffusion module employs a 3D face model to
generate face sketch from text description. Experiments on the FFHQ dataset
show the effectiveness of our method. We will release our dataset and code for
future research.

本文提出了一种新颖的可控凝视的人脸生成任务，方法使用文本描述输入来生成具有相应凝视特征的人脸图像，并通过面部草图和 3D 人脸模型实现了凝视的文本到人脸的转换。实验证明了方法的有效性，同时作者将提供数据集和代码以供今后研究使用。