We introduce Multi-view Ancestral Sampling (MAS), a method for generating
consistent multi-view 2D samples of a motion sequence, enabling the creation of
its 3D counterpart. MAS leverages a diffusion model trained solely on 2D data,
opening opportunities to exciting and diverse fields of motion previously
under-explored as 3D data is scarce and hard to collect. MAS works by
simultaneously denoising multiple 2D motion sequences representing the same
motion from different angles. Our consistency block ensures consistency across
all views at each diffusion step by combining the individual generations into a
unified 3D sequence, and projecting it back to the original views for the next
iteration. We demonstrate MAS on 2D pose data acquired from videos depicting
professional basketball maneuvers, rhythmic gymnastic performances featuring a
ball apparatus, and horse obstacle course races. In each of these domains, 3D
motion capture is arduous, and yet, MAS generates diverse and realistic 3D
sequences without textual conditioning. As we demonstrate, our ancestral
sampling-based approach offers a more natural integration with the diffusion
framework compared to popular denoising optimization-based approaches, and
avoids common issues such as out-of-domain sampling, lack of details and
mode-collapse. this https URL

我们介绍了一种名为多视角祖先采样（MAS）的方法，用于生成一系列运动的一致的多视角 2D 样本，从而实现其 3D 对应物的创建。MAS 利用仅基于 2D 数据训练的扩散模型，为 3D 数据稀缺且难以收集的运动领域提供了机会。MAS 通过同时去噪不同角度下表示相同运动的多个 2D 运动序列，保证了每个扩散步骤中所有视角的一致性，并将个体生成结果合并为统一的 3D 序列，再投影到原始视角作为下一迭代的输入。我们在包括职业篮球动作、球体器械韵律体操表演和马障碍赛等视频中的 2D 姿势数据上展示了 MAS 的效果。在这些领域中，进行 3D 运动捕捉工作较为困难，然而 MAS 却能生成多样且逼真的 3D 序列，无需文本约束。正如我们展示的，我们基于祖先采样的方法相对于流行的基于优化的去噪方法，在扩散框架中提供了更自然的一体化，避免了常见的问题，如域外采样、缺乏细节和模式坍缩。