We present the first method capable of photorealistically reconstructing
deformable scenes using photos/videos captured casually from mobile phones. Our
approach augments neural radiance fields (NeRF) by optimizing an additional
continuous volumetric deformation field that warps each observed point into a
canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone
to local minima, and propose a coarse-to-fine optimization method for
coordinate-based models that allows for more robust optimization. By adapting
principles from geometry processing and physical simulation to NeRF-like
models, we propose an elastic regularization of the deformation field that
further improves robustness. We show that our method can turn casually captured
selfie photos/videos into deformable NeRF models that allow for photorealistic
renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We
evaluate our method by collecting time-synchronized data using a rig with two
mobile phones, yielding train/validation images of the same pose at different
viewpoints. We show that our method faithfully reconstructs non-rigidly
deforming scenes and reproduces unseen views with high fidelity.

描述了一种基于神经辐射场的方法，通过优化连续的体形变场使移动设备拍摄的照片 / 视频重建变形场景，旨在生成高度逼真的渲染图像。实验表明，该方法可使用手机拍摄的自拍照片 / 视频转换成可变形的 NeRF 模型，并在任意视点产生高保真度的视图。