Face-to-face communication is a common scenario including roles of speakers
and listeners. Most existing research methods focus on producing speaker
videos, while the generation of listener heads remains largely overlooked.
Responsive listening head generation is an important task that aims to model
face-to-face communication scenarios by generating a listener head video given
a speaker video and a listener head image. An ideal generated responsive
listening video should respond to the speaker with attitude or viewpoint
expressing while maintaining diversity in interaction patterns and accuracy in
listener identity information. To achieve this goal, we propose the
\textbf{M}ulti-\textbf{F}aceted \textbf{R}esponsive Listening Head Generation
Network (MFR-Net). Specifically, MFR-Net employs the probabilistic denoising
diffusion model to predict diverse head pose and expression features. In order
to perform multi-faceted response to the speaker video, while maintaining
accurate listener identity preservation, we design the Feature Aggregation
Module to boost listener identity features and fuse them with other
speaker-related features. Finally, a renderer finetuned with identity
consistency loss produces the final listening head videos. Our extensive
experiments demonstrate that MFR-Net not only achieves multi-faceted responses
in diversity and speaker identity information but also in attitude and
viewpoint expression.

响应式听觉头部生成网络（MFR-Net）旨在通过生成侦听视频来模拟面对面通信情境，以姿态和观点表达对发言人作出响应，同时保持交互模式的多样性和侦听者身份信息的准确性。

多方面响应式听力头生成的 MFR-Net：通过去噪扩散模型

MFR-Net: Multi-faceted Responsive Listening Head Generation via  Denoising Diffusion Model

Gestures are integral components of face-to-face communication. They unfold
over time, often following predictable movement phases of preparation, stroke,
and retraction. Yet, the prevalent approach to automatic gesture detection
treats the problem as binary classification, classifying a segment as either
containing a gesture or not, thus failing to capture its inherently sequential
and contextual nature. To address this, we introduce a novel framework that
reframes the task as a multi-phase sequence labeling problem rather than binary
classification. Our model processes sequences of skeletal movements over time
windows, uses Transformer encoders to learn contextual embeddings, and
leverages Conditional Random Fields to perform sequence labeling. We evaluate
our proposal on a large dataset of diverse co-speech gestures in task-oriented
face-to-face dialogues. The results consistently demonstrate that our method
significantly outperforms strong baseline models in detecting gesture strokes.
Furthermore, applying Transformer encoders to learn contextual embeddings from
movement sequences substantially improves gesture unit detection. These results
highlight our framework's capacity to capture the fine-grained dynamics of
co-speech gesture phases, paving the way for more nuanced and accurate gesture
detection and analysis.

我们提出了一种新的框架，将手势检测任务看作多阶段序列标注问题，通过处理时间窗口内的骨骼运动序列，使用 Transformer 编码器学习上下文嵌入，并利用条件随机场进行序列标注。我们在大规模数据集上对我们的方法进行评估，结果表明，我们的方法在检测手势笔划方面明显优于强基准模型。此外，将 Transformer 编码器应用于从运动序列中学习上下文嵌入显著改善了手势单元检测。这些结果突显了我们框架捕捉共话手势阶段微观动态的能力，为更细致和准确的手势检测和分析铺平了道路。

通过多阶段序列标注实现共喃手势检测

Co-Speech Gesture Detection through Multi-phase Sequence Labeling

Human face-to-face communication is a complex multimodal signal. We use words
(language modality), gestures (vision modality) and changes in tone (acoustic
modality) to convey our intentions. Humans easily process and understand
face-to-face communication, however, comprehending this form of communication
remains a significant challenge for Artificial Intelligence (AI). AI must
understand each modality and the interactions between them that shape human
communication. In this paper, we present a novel neural architecture for
understanding human communication called the Multi-attention Recurrent Network
(MARN). The main strength of our model comes from discovering interactions
between modalities through time using a neural component called the
Multi-attention Block (MAB) and storing them in the hybrid memory of a
recurrent component called the Long-short Term Hybrid Memory (LSTHM). We
perform extensive comparisons on six publicly available datasets for multimodal
sentiment analysis, speaker trait recognition and emotion recognition. MARN
shows state-of-the-art performance on all the datasets.

本文介绍了一种名为多注意力循环网络（MARN）的人类交流理解神经架构，通过使用多注意力块（MAB）的神经组件，MARN 致力于发现和存储影响人际交往的互动，并在多模态情感分析、说话者特质识别和情感识别方面显示出最先进的性能。