In dyadic interactions, humans communicate their intentions and state of mind
using verbal and non-verbal cues, where multiple different facial reactions
might be appropriate in response to a specific speaker behaviour. Then, how to
develop a machine learning (ML) model that can automatically generate multiple
appropriate, diverse, realistic and synchronised human facial reactions from an
previously unseen speaker behaviour is a challenging task. Following the
successful organisation of the first REACT challenge (REACT 2023), this edition
of the challenge (REACT 2024) employs a subset used by the previous challenge,
which contains segmented 30-secs dyadic interaction clips originally recorded
as part of the NOXI and RECOLA datasets, encouraging participants to develop
and benchmark Machine Learning (ML) models that can generate multiple
appropriate facial reactions (including facial image sequences and their
attributes) given an input conversational partner's stimulus under various
dyadic video conference scenarios. This paper presents: (i) the guidelines of
the REACT 2024 challenge; (ii) the dataset utilized in the challenge; and (iii)
the performance of the baseline systems on the two proposed sub-challenges:
Offline Multiple Appropriate Facial Reaction Generation and Online Multiple
Appropriate Facial Reaction Generation, respectively. The challenge baseline
code is publicly available at
this https URL

通过机器学习模型在 REACT 2024 挑战中开发和基准测试了能够生成多个适当、多样、真实和同步的面部反应的方法，并提供了该挑战的指南、数据集以及在两个子挑战中基线系统的性能。

REACT 2024：第二届多目标适应性面部反应生成挑战

REACT 2024: the Second Multiple Appropriate Facial Reaction Generation  Challenge

In dyadic interaction, predicting the listener's facial reactions is
challenging as different reactions may be appropriate in response to the same
speaker's behaviour. This paper presents a novel framework called ReactFace
that learns an appropriate facial reaction distribution from a speaker's
behaviour rather than replicating the real facial reaction of the listener.
ReactFace generates multiple different but appropriate photo-realistic human
facial reactions by (i) learning an appropriate facial reaction distribution
representing multiple appropriate facial reactions; and (ii) synchronizing the
generated facial reactions with the speaker's verbal and non-verbal behaviours
at each time stamp, resulting in realistic 2D facial reaction sequences.
Experimental results demonstrate the effectiveness of our approach in
generating multiple diverse, synchronized, and appropriate facial reactions
from each speaker's behaviour, with the quality of the generated reactions
being influenced by the speaker's speech and facial behaviours. Our code is
made publicly available at https://github.com/lingjivoo/ReactFace.

本文提出了一种名为 ReactFace 的新框架，通过学习代表多种适当面部反应的面部反应分布，并将生成的面部反应与说话者的语言和非语言行为同步，生成多个不同但适当的逼真的人类面部反应序列。

ReactFace: 双向交互中多样化恰当的面部表情生成

ReactFace: Multiple Appropriate Facial Reaction Generation in Dyadic  Interactions

Generating facial reactions in a human-human dyadic interaction is complex
and highly dependent on the context since more than one facial reactions can be
appropriate for the speaker's behaviour. This has challenged existing machine
learning (ML) methods, whose training strategies enforce models to reproduce a
specific (not multiple) facial reaction from each input speaker behaviour. This
paper proposes the first multiple appropriate facial reaction generation
framework that re-formulates the one-to-many mapping facial reaction generation
problem as a one-to-one mapping problem. This means that we approach this
problem by considering the generation of a distribution of the listener's
appropriate facial reactions instead of multiple different appropriate facial
reactions, i.e., 'many' appropriate facial reaction labels are summarised as
'one' distribution label during training. Our model consists of a perceptual
processor, a cognitive processor, and a motor processor. The motor processor is
implemented with a novel Reversible Multi-dimensional Edge Graph Neural Network
(REGNN). This allows us to obtain a distribution of appropriate real facial
reactions during the training process, enabling the cognitive processor to be
trained to predict the appropriate facial reaction distribution. At the
inference stage, the REGNN decodes an appropriate facial reaction by using this
distribution as input. Experimental results demonstrate that our approach
outperforms existing models in generating more appropriate, realistic, and
synchronized facial reactions. The improved performance is largely attributed
to the proposed appropriate facial reaction distribution learning strategy and
the use of a REGNN. The code is available at
this https URL

本文提出了第一个多恰当面部反应生成框架，通过将一对多映射面部反应生成问题重新转换为一个对一个映射问题，提出适当的面部反应分布学习策略，利用新颖的可逆多维边缘图神经网络 (REGNN) 解码恰当的面部反应，实验结果表明，利用该方法生成的面部反应更为恰当、逼真和同步