Recent advances in imitation learning for 3D robotic manipulation have shown promising results with diffusion-based policies. However, achieving human-level dexterity requires seamless integration of geometric precision and semantic understanding. We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D semantic representation by leveraging foundation models. Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates. This integration enables complete semantic understanding even under occlusions while eliminating manual annotation requirements. By incorporating semantic flow into diffusion policies, we demonstrate significant improvements in both terminal-constrained manipulation and cross-object generalization. Extensive experiments across five simulation tasks show that G3Flow consistently outperforms existing approaches, achieving up to 68.3% and 50.1% average success rates on terminal-constrained manipulation and cross-object generalization tasks respectively. Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies.

该研究解决了现有3D机器人操作中缺乏几何精度与语义理解无缝集成的问题。提出的G3Flow框架通过实时构建动态的物体中心三维语义流，结合3D生成模型和视觉基础模型，实现了在遮挡条件下的完整语义理解及跨物体的泛化能力。通过将语义流融入扩散策略，该方法在终端限制操作和跨物体泛化任务中明显提高了成功率，展示了其在增强机器人操作政策中的潜力。

G3Flow：面向姿态感知和可泛化物体操作的生成性三维语义流