Face-to-face communication is a common scenario including roles of speakers
and listeners. Most existing research methods focus on producing speaker
videos, while the generation of listener heads remains largely overlooked.
Responsive listening head generation is an important task that aims to model
face-to-face communication scenarios by generating a listener head video given
a speaker video and a listener head image. An ideal generated responsive
listening video should respond to the speaker with attitude or viewpoint
expressing while maintaining diversity in interaction patterns and accuracy in
listener identity information. To achieve this goal, we propose the
\textbf{M}ulti-\textbf{F}aceted \textbf{R}esponsive Listening Head Generation
Network (MFR-Net). Specifically, MFR-Net employs the probabilistic denoising
diffusion model to predict diverse head pose and expression features. In order
to perform multi-faceted response to the speaker video, while maintaining
accurate listener identity preservation, we design the Feature Aggregation
Module to boost listener identity features and fuse them with other
speaker-related features. Finally, a renderer finetuned with identity
consistency loss produces the final listening head videos. Our extensive
experiments demonstrate that MFR-Net not only achieves multi-faceted responses
in diversity and speaker identity information but also in attitude and
viewpoint expression.

响应式听觉头部生成网络（MFR-Net）旨在通过生成侦听视频来模拟面对面通信情境，以姿态和观点表达对发言人作出响应，同时保持交互模式的多样性和侦听者身份信息的准确性。

多方面响应式听力头生成的 MFR-Net：通过去噪扩散模型

MFR-Net: Multi-faceted Responsive Listening Head Generation via  Denoising Diffusion Model

We introduce a new conversation head generation benchmark for synthesizing
behaviors of a single interlocutor in a face-to-face conversation. The
capability to automatically synthesize interlocutors which can participate in
long and multi-turn conversations is vital and offer benefits for various
applications, including digital humans, virtual agents, and social robots.
While existing research primarily focuses on talking head generation (one-way
interaction), hindering the ability to create a digital human for conversation
(two-way) interaction due to the absence of listening and interaction parts. In
this work, we construct two datasets to address this issue, ``ViCo'' for
independent talking and listening head generation tasks at the sentence level,
and ``ViCo-X'', for synthesizing interlocutors in multi-turn conversational
scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the
interaction modeling during the face-to-face conversation: 1) responsive
listening head generation making listeners respond actively to the speaker with
non-verbal signals, 2) expressive talking head generation guiding speakers to
be aware of listeners' behaviors, and 3) conversational head generation to
integrate the talking/listening ability in one interlocutor. Along with the
datasets, we also propose corresponding baseline solutions to the three
aforementioned tasks. Experimental results show that our baseline method could
generate responsive and vivid agents that can collaborate with real person to
fulfil the whole conversation. Project page: this https URL

该研究提出了一种新的对话头生成基准，用于在面对面对话中合成单个交流者的行为，通过构建两个数据集，并针对面对面对话中的互动建立三个新任务，我们定义了相应的基线解决方案，并通过实验结果表明我们的基线方法可以生成具有响应能力和生动感的代理人，可以与真人协作完成整个对话。