Toward unlocking the potential of generative models in immersive 4D
experiences, we introduce Virtual Pet, a novel pipeline to model realistic and
diverse motions for target animal species within a 3D environment. To
circumvent the limited availability of 3D motion data aligned with
environmental geometry, we leverage monocular internet videos and extract
deformable NeRF representations for the foreground and static NeRF
representations for the background. For this, we develop a reconstruction
strategy, encompassing species-level shared template learning and per-video
fine-tuning. Utilizing the reconstructed data, we then train a conditional 3D
motion model to learn the trajectory and articulation of foreground animals in
the context of 3D backgrounds. We showcase the efficacy of our pipeline with
comprehensive qualitative and quantitative evaluations using cat videos. We
also demonstrate versatility across unseen cats and indoor environments,
producing temporally coherent 4D outputs for enriched virtual experiences.

通过 Virtual Pet 管道，我们介绍了一种解锁生成模型在沉浸式 4D 体验中潜力的方法，用于在 3D 环境中为目标动物物种建模真实多样的动作。通过利用单眼互联网视频并提取可变形 NeRF 表示前景和静态 NeRF 表示背景来规避与环境几何对齐的 3D 动作数据的有限可用性。我们展示了我们的管道的有效性，使用猫视频进行了全面的定性和定量评估，并表明在未见过的猫和室内环境中也具有多样性，为丰富的虚拟体验产生了时间上连续的 4D 输出。