In this study, our goal is to create interactive avatar agents that can autonomously plan and animate nuanced facial movements realistically, from both visual and behavioral perspectives. Given high-level inputs about the environment and agent profile, our framework harnesses LLMs to produce a series of detailed text descriptions of the avatar agents' facial motions. These descriptions are then processed by our task-agnostic driving engine into motion token sequences, which are subsequently converted into continuous motion embeddings that are further consumed by our standalone neural-based renderer to generate the final photorealistic avatar animations. These streamlined processes allow our framework to adapt to a variety of non-verbal avatar interactions, both monadic and dyadic. Our extensive study, which includes experiments on both newly compiled and existing datasets featuring two types of agents -- one capable of monadic interaction with the environment, and the other designed for dyadic conversation -- validates the effectiveness and versatility of our approach. To our knowledge, we advanced a leap step by combining LLMs and neural rendering for generalized non-verbal prediction and photo-realistic rendering of avatar agents.

我们的研究目标是创建可以自主规划和模拟细致逼真的面部动作的交互式化身代理机器人，从视觉和行为角度进行研究。我们的研究通过将大规模语言模型应用到化身代理机器人面部动作描述生成中，再通过神经渲染器生成最终逼真的化身动画，并经过实验证实了这一方法的有效性和通用性。

AgentAvatar: 解耦规划、驾驶和渲染以实现逼真的虚拟人物代理