While considerable progress has been made in achieving accurate lip
synchronization for 3D speech-driven talking face generation, the task of
incorporating expressive facial detail synthesis aligned with the speaker's
speaking status remains challenging. Our goal is to directly leverage the
inherent style information conveyed by human speech for generating an
expressive talking face that aligns with the speaking status. In this paper, we
propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking
face generation. This system harnesses the robust contextual reasoning and
hallucination capability offered by Large Language Models (LLMs) to instruct
the realistic synthesis of 3D talking faces. Instead of directly learning
facial movements from human speech, our two-stage strategy involves the LLMs
first comprehending audio information and generating instructions implying
expressive facial details seamlessly corresponding to the speech. Subsequently,
a diffusion-based generative network executes these instructions. This
two-stage process, coupled with the incorporation of LLMs, enhances model
interpretability and provides users with flexibility to comprehend instructions
and specify desired operations or modifications. Extensive experiments showcase
the effectiveness of our approach in producing vivid talking faces with
expressive facial movements and consistent emotional status.

利用大型语言模型指导实现具有表情细节合成能力的说话人脸生成系统，通过先理解语音信息并生成指令，再执行这些指令生成具有表情运动的生动说话人脸，实验证明该方法有效且具有一致的情感状态。