The field of neural rendering has witnessed significant progress with advancements in generative models and differentiable rendering techniques. Though 2D diffusion has achieved success, a unified 3D diffusion pipeline remains unsettled. This paper introduces a novel framework called LN3Diff to address this gap and enable fast, high-quality, and generic conditional 3D generation. Our approach harnesses a 3D-aware architecture and variational autoencoder (VAE) to encode the input image into a structured, compact, and 3D latent space. The latent is decoded by a transformer-based decoder into a high-capacity 3D neural field. Through training a diffusion model on this 3D-aware latent space, our method achieves state-of-the-art performance on ShapeNet for 3D generation and demonstrates superior performance in monocular 3D reconstruction and conditional 3D generation across various datasets. Moreover, it surpasses existing 3D diffusion methods in terms of inference speed, requiring no per-instance optimization. Our proposed LN3Diff presents a significant advancement in 3D generative modeling and holds promise for various applications in 3D vision and graphics tasks.

LN3Diff是一个新颖的框架，通过使用3D感知架构和变分自编码器将输入图像编码到结构化、紧凑和3D潜空间中，然后通过基于Transformer的解码器将潜空间解码为高容量的3D神经场，从而在3D生成、单目3D重建和有条件3D生成等各种数据集上展现出卓越的表现，并在推理速度方面超越了现有的3D扩散方法。

LN3Diff：可扩展的潜在神经场扩散用于高速3D生成