We present a framework for modeling interactional communication in dyadic
conversations: given multimodal inputs of a speaker, we autoregressively output
multiple possibilities of corresponding listener motion. We combine the motion
and speech audio of the speaker using a motion-audio cross attention
transformer. Furthermore, we enable non-deterministic prediction by learning a
discrete latent representation of realistic listener motion with a novel
motion-encoding VQ-VAE. Our method organically captures the multimodal and
non-deterministic nature of nonverbal dyadic interactions. Moreover, it
produces realistic 3D listener facial motion synchronous with the speaker (see
video). We demonstrate that our method outperforms baselines qualitatively and
quantitatively via a rich suite of experiments. To facilitate this line of
research, we introduce a novel and large in-the-wild dataset of dyadic
conversations. Code, data, and videos available at
this https URL

该论文提出了一个交互式对话的建模框架，通过结合说话人的动作、对话音频和可能的相应听众动作输出来捕捉非口头双人互动的多模态和非确定性特征，还使用了运动 - 音频交叉注意力变换将说话人的运动和语音音频结合起来，并通过运动编码 VQ-VAE 学习了对真实听众运动的离散潜在表示。