Socially Interactive Agents (SIAs) are physical or virtual embodied agents
that display similar behavior as human multimodal behavior. Modeling SIAs'
non-verbal behavior, such as speech and facial gestures, has always been a
challenging task, given that a SIA can take the role of a speaker or a
listener. A SIA must emit appropriate behavior adapted to its own speech, its
previous behaviors (intra-personal), and the User's behaviors (inter-personal)
for both roles. We propose AMII, a novel approach to synthesize adaptive facial
gestures for SIAs while interacting with Users and acting interchangeably as a
speaker or as a listener. AMII is characterized by modality memory encoding
schema - where modality corresponds to either speech or facial gestures - and
makes use of attention mechanisms to capture the intra-personal and
inter-personal relationships. We validate our approach by conducting objective
evaluations and comparing it with the state-of-the-art approaches.

提出 AMII 方法，利用注意力机制综合非言语行为以适应 Socially Interactive Agents 的不同角色。

AMII: 自适应多模态人际和自我模型用于行为合成

AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for  Adapted Behavior Synthesis

The effectiveness of computerized cognitive training in slowing cognitive
decline and brain aging in dementia is often limited by the engagement of
participants in the training. Monitoring older users' real-time engagement in
domains of attention, motivation, and affect is crucial to understanding the
overall effectiveness of such training. In this paper, we propose to predict
engagement, quantified via an established mental fatigue measure assessing
users' perceived attention, motivation, and affect throughout computerized
cognitive training sessions, in older adults with mild cognitive impairment
(MCI), by monitoring their real-time video-recorded facial gestures in training
sessions. To achieve the goal, we used computer vision, analyzing video frames
every 5 seconds to optimize the balance between information retention and data
size, and developed a novel Recurrent Video Transformer (RVT). Our RVT model,
which combines a clip-wise transformer encoder module and a session-wise
Recurrent Neural Network (RNN) classifier, achieved the highest balanced
accuracy, F1 score, and precision compared to other state-of-the-art models for
both detecting mental fatigue/disengagement cases (binary classification) and
rating the level of mental fatigue (multi-class classification). By leveraging
dynamic temporal information, the RVT model demonstrates the potential to
accurately predict engagement among computerized cognitive training users,
which lays the foundation for future work to modulate the level of engagement
in computerized cognitive training interventions. The code will be released.

本论文利用面部表情分析技术，运用回归视频转换（RVT）模型对老年轻度认知障碍患者的实时动态监测，实现对认知训练过程中注意力、动机和情感等方面的可持续参与度的评估，该模型根据视频帧分析进行计算，能够精确预测认知训练中的参与度，对于之后的计算机认知训练干预工作奠定了基础。

基于视觉的循环 Transformer 编码器用于认知训练会话中的疲劳和参与度估计

Recurrent Transformer Encoders for Vision-based Estimation of Fatigue  and Engagement in Cognitive Training Sessions

Neural networks have recently become good at engaging in dialog. However,
current approaches are based solely on verbal text, lacking the richness of a
real face-to-face conversation. We propose a neural conversation model that
aims to read and generate facial gestures alongside with text. This allows our
model to adapt its response based on the "mood" of the conversation. In
particular, we introduce an RNN encoder-decoder that exploits the movement of
facial muscles, as well as the verbal conversation. The decoder consists of two
layers, where the lower layer aims at generating the verbal response and coarse
facial expressions, while the second layer fills in the subtle gestures, making
the generated output more smooth and natural. We train our neural network by
having it "watch" 250 movies. We showcase our joint face-text model in
generating more natural conversations through automatic metrics and a human
study. We demonstrate an example application with a face-to-face chatting
avatar.

我们提出了一个新的神经会话模型，其不仅读取和生成文本信息，还能够识别和生成面部表情，以及基于对话中的情绪适应其响应。该模型采用 RNN 编码器 - 解码器架构，通过观察电影进行训练，并通过自动度量和人类研究展示了其生成更自然对话的能力。