Diffusion Transformers have recently shown remarkable effectiveness in
generating high-quality 3D point clouds. However, training voxel-based
diffusion models for high-resolution 3D voxels remains prohibitively expensive
due to the cubic complexity of attention operators, which arises from the
additional dimension of voxels. Motivated by the inherent redundancy of 3D
compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
tailored for efficient 3D point cloud generation, which greatly reduces
training costs. Specifically, we draw inspiration from masked autoencoders to
dynamically operate the denoising process on masked voxelized point clouds. We
also propose a novel voxel-aware masking strategy to adaptively aggregate
background/foreground information from voxelized point clouds. Our method
achieves state-of-the-art performance with an extreme masking ratio of nearly
99%. Moreover, to improve multi-category 3D generation, we introduce
Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
distinct diffusion path with different experts, relieving gradient conflict.
Experimental results on the ShapeNet dataset demonstrate that our method
achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
metrics when generating 128-resolution voxel point clouds, using only 6.5% of
the original training cost.

FastDiT-3D is a novel masked diffusion transformer designed for efficient generation of high-quality 3D point clouds, achieving state-of-the-art performance with reduced training costs and improving multi-category 3D generation using a Mixture-of-Expert approach.

快速训练极端掩码的扩散变压器用于三维点云生成

Fast Training of Diffusion Transformer with Extreme Masking for 3D Point  Clouds Generation

End-to-end models with large capacity have significantly improved
multilingual automatic speech recognition, but their computation cost poses
challenges for on-device applications. We propose a streaming truly
multilingual Conformer incorporating mixture-of-expert (MoE) layers that learn
to only activate a subset of parameters in training and inference. The MoE
layer consists of a softmax gate which chooses the best two experts among many
in forward propagation. The proposed MoE layer offers efficient inference by
activating a fixed number of parameters as the number of experts increases. We
evaluate the proposed model on a set of 12 languages, and achieve an average
11.9% relative improvement in WER over the baseline. Compared to an adapter
model using ground truth information, our MoE model achieves similar WER and
activates similar number of parameters but without any language information. We
further show around 3% relative WER improvement by multilingual shallow fusion.

提出了一种流式的、真正的多语言 Conformer 模型，其中融合了 MoE 层以提高推理效率，经在 12 种语言上评估，相对于基线获得 11.9% 的相对 WER 改进；在不使用语言信息的前提下，与使用 Ground Truth 信息的适配器模型相比，我们的 MoE 模型实现了类似的 WER 和相似数量的参数激活，并通过多语言浅层融合实现了约 3% 的相对 WER 改进。