This paper presents DNAct, a language-conditioned multi-task policy framework
that integrates neural rendering pre-training and diffusion training to enforce
multi-modality learning in action sequence spaces. To learn a generalizable
multi-task policy with few demonstrations, the pre-training phase of DNAct
leverages neural rendering to distill 2D semantic features from foundation
models such as Stable Diffusion to a 3D space, which provides a comprehensive
semantic understanding regarding the scene. Consequently, it allows various
applications to challenging robotic tasks requiring rich 3D semantics and
accurate geometry. Furthermore, we introduce a novel approach utilizing
diffusion training to learn a vision and language feature that encapsulates the
inherent multi-modality in the multi-task demonstrations. By reconstructing the
action sequences from different tasks via the diffusion process, the model is
capable of distinguishing different modalities and thus improving the
robustness and the generalizability of the learned representation. DNAct
significantly surpasses SOTA NeRF-based multi-task manipulation approaches with
over 30% improvement in success rate. Project website: dnact.github.io.

本文介绍了 DNAct，一种语言驱动的多任务策略框架。该框架集成了神经渲染预训练和扩散训练，以在动作序列空间中实现多模态学习。通过利用神经渲染从基础模型中提取二维语义特征并转化到三维空间，DNAct 可以对场景进行全面的语义理解，从而适用于需要丰富三维语义和准确几何信息的各种挑战性机器人任务。此外，通过扩散训练学习视觉和语言特征， DNAct 能够区分不同任务的多模态特征从而提高学习表示的鲁棒性和泛化性能。与现有的基于 NeRF 的多任务操作方法相比，DNAct 在成功率上有 30% 以上的改进。