We introduce a new conversation head generation benchmark for synthesizing
behaviors of a single interlocutor in a face-to-face conversation. The
capability to automatically synthesize interlocutors which can participate in
long and multi-turn conversations is vital and offer benefits for various
applications, including digital humans, virtual agents, and social robots.
While existing research primarily focuses on talking head generation (one-way
interaction), hindering the ability to create a digital human for conversation
(two-way) interaction due to the absence of listening and interaction parts. In
this work, we construct two datasets to address this issue, ``ViCo'' for
independent talking and listening head generation tasks at the sentence level,
and ``ViCo-X'', for synthesizing interlocutors in multi-turn conversational
scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the
interaction modeling during the face-to-face conversation: 1) responsive
listening head generation making listeners respond actively to the speaker with
non-verbal signals, 2) expressive talking head generation guiding speakers to
be aware of listeners' behaviors, and 3) conversational head generation to
integrate the talking/listening ability in one interlocutor. Along with the
datasets, we also propose corresponding baseline solutions to the three
aforementioned tasks. Experimental results show that our baseline method could
generate responsive and vivid agents that can collaborate with real person to
fulfil the whole conversation. Project page: this https URL

该研究提出了一种新的对话头生成基准，用于在面对面对话中合成单个交流者的行为，通过构建两个数据集，并针对面对面对话中的互动建立三个新任务，我们定义了相应的基线解决方案，并通过实验结果表明我们的基线方法可以生成具有响应能力和生动感的代理人，可以与真人协作完成整个对话。

交互式对话式头像生成

Interactive Conversational Head Generation

We present a new listening head generation benchmark, for synthesizing
responsive feedbacks of a listener (e.g., nod, smile) during a face-to-face
conversation. As the indispensable complement to talking heads generation,
listening head generation has seldomly been studied in literature.
Automatically synthesizing listening behavior that actively responds to a
talking head, is critical to applications such as digital human, virtual agents
and social robots. In this work, we propose a novel dataset "ViCo",
highlighting the listening head generation during a face-to-face conversation.
A total number of 92 identities (67 speakers and 76 listeners) are involved in
ViCo, featuring 483 clips in a paired "speaking-listening" pattern, where
listeners show three listening styles based on their attitudes: positive,
neutral, negative. Different from traditional speech-to-gesture or talking-head
generation, listening head generation takes as input both the audio and visual
signals from the speaker, and gives non-verbal feedbacks (e.g., head motions,
facial expressions) in a real-time manner. Our dataset supports a wide range of
applications such as human-to-human interaction, video-to-video translation,
cross-modal understanding and generation. To encourage further research, we
also release a listening head generation baseline, conditioning on different
listening attitudes. Code & ViCo dataset: this https URL

本文介绍了一种新的人脸 - 面对面谈话场景下，回应听众的反馈（如点头、微笑）的生成模型，并针对此场景提出了新的数据集 “ViCo”，该数据集支持一系列应用，如人机交互、视频翻译和跨模态生成，同时提供了生成模型的基线。

响应式听觉头部生成：一个基准数据集和基准模型

Responsive Listening Head Generation: A Benchmark Dataset and Baseline

We present Neural Head Avatars, a novel neural representation that explicitly
models the surface geometry and appearance of an animatable human avatar that
can be used for teleconferencing in AR/VR or other applications in the movie or
games industry that rely on a digital human. Our representation can be learned
from a monocular RGB portrait video that features a range of different
expressions and views. Specifically, we propose a hybrid representation
consisting of a morphable model for the coarse shape and expressions of the
face, and two feed-forward networks, predicting vertex offsets of the
underlying mesh as well as a view- and expression-dependent texture. We
demonstrate that this representation is able to accurately extrapolate to
unseen poses and view points, and generates natural expressions while providing
sharp texture details. Compared to previous works on head avatars, our method
provides a disentangled shape and appearance model of the complete human head
(including hair) that is compatible with the standard graphics pipeline.
Moreover, it quantitatively and qualitatively outperforms current state of the
art in terms of reconstruction quality and novel-view synthesis.

我们提出了神经头像，该方法能够显式建模动画人物头像的表面几何形状和外观，是虚拟现实和其他电影或游戏应用中数字人物的一种有效表示方法，它能够从单目 RGB 人像视频中学习，能够精确地外推到未知姿态和观察角度，并产生自然的表情和清晰的纹理细节。