Dynamic NeRFs have recently garnered growing attention for 3D talking
portrait synthesis. Despite advances in rendering speed and visual quality,
challenges persist in enhancing efficiency and effectiveness. We present
R2-Talker, an efficient and effective framework enabling realistic real-time
talking head synthesis. Specifically, using multi-resolution hash grids, we
introduce a novel approach for encoding facial landmarks as conditional
features. This approach losslessly encodes landmark structures as conditional
features, decoupling input diversity, and conditional spaces by mapping
arbitrary landmarks to a unified feature space. We further propose a scheme of
progressive multilayer conditioning in the NeRF rendering pipeline for
effective conditional feature fusion. Our new approach has the following
advantages as demonstrated by extensive experiments compared with the
state-of-the-art works: 1) The lossless input encoding enables acquiring more
precise features, yielding superior visual quality. The decoupling of inputs
and conditional spaces improves generalizability. 2) The fusing of conditional
features and MLP outputs at each MLP layer enhances conditional impact,
resulting in more accurate lip synthesis and better visual quality. 3) It
compactly structures the fusion of conditional features, significantly
enhancing computational efficiency.

R2-Talker 是一个高效和有效的框架，可以实现逼真的实时说话头部合成，通过使用多分辨率哈希网格来编码面部标志作为条件特征，并在 NeRF 渲染管道中引入渐进多层条件化方案，优化了条件特征融合、嘴巴合成和视觉质量。