We introduce Animate124 (Animate-one-image-to-4D), the first work to animate
a single in-the-wild image into 3D video through textual motion descriptions,
an underexplored problem with significant applications. Our 4D generation
leverages an advanced 4D grid dynamic Neural Radiance Field (NeRF) model,
optimized in three distinct stages using multiple diffusion priors. Initially,
a static model is optimized using the reference image, guided by 2D and 3D
diffusion priors, which serves as the initialization for the dynamic NeRF.
Subsequently, a video diffusion model is employed to learn the motion specific
to the subject. However, the object in the 3D videos tends to drift away from
the reference image over time. This drift is mainly due to the misalignment
between the text prompt and the reference image in the video diffusion model.
In the final stage, a personalized diffusion prior is therefore utilized to
address the semantic drift. As the pioneering image-text-to-4D generation
framework, our method demonstrates significant advancements over existing
baselines, evidenced by comprehensive quantitative and qualitative assessments.

通过文本动作描述将野外的单幅图像动画化为 3D 视频是我们引入的 Animate124（一个图像到 4D 的动画）的第一个工作，这是一个尚未深入探索但具有重要应用的问题。我们的 4D 生成利用了先进的 4D 网格动态神经辐射场（NeRF）模型，在三个不同阶段使用多个扩散先验进行优化。

Animate124：一个将一张静态图片转化为 4D 动态场景的动画技术

Animate124: Animating One Image to 4D Dynamic Scene

Recent years have witnessed a rapid development of immersive multimedia which
bridges the gap between the real world and virtual space. Volumetric videos, as
an emerging representative 3D video paradigm that empowers extended reality,
stand out to provide unprecedented immersive and interactive video watching
experience. Despite the tremendous potential, the research towards 3D
volumetric video is still in its infancy, relying on sufficient and complete
datasets for further exploration. However, existing related volumetric video
datasets mostly only include a single object, lacking details about the scene
and the interaction between them. In this paper, we focus on the current most
widely used data format, point cloud, and for the first time release a
full-scene volumetric video dataset that includes multiple people and their
daily activities interacting with the external environments. Comprehensive
dataset description and analysis are conducted, with potential usage of this
dataset. The dataset and additional tools can be accessed via the following
website: this https URL

本文描述和分析了第一个包含多个人和场景互动的全景体数据集并附加数据工具

FSVVD: 一个全场景体积视频数据集

FSVVD: A Dataset of Full Scene Volumetric Video

Generative models have emerged as an essential building block for many image
synthesis and editing tasks. Recent advances in this field have also enabled
high-quality 3D or video content to be generated that exhibits either
multi-view or temporal consistency. With our work, we explore 4D generative
adversarial networks (GANs) that learn unconditional generation of 3D-aware
videos. By combining neural implicit representations with time-aware
discriminator, we develop a GAN framework that synthesizes 3D video supervised
only with monocular videos. We show that our method learns a rich embedding of
decomposable 3D structures and motions that enables new visual effects of
spatio-temporal renderings while producing imagery with quality comparable to
that of existing 3D or video GANs.

本文探讨了生成对抗网络 (GANs)，将神经隐式表示与时间感知区别器相结合，发展了一种 GAN 框架，仅用单目视频合成了 3D 视频，能够学习可分解的 3D 结构和运动的丰富嵌入，实现了新的时空渲染视觉效果，同时也能产生与现有 3D 或视频 GANs 相媲美的图像质量。