Performing language-conditioned robotic manipulation tasks in unstructured
environments is highly demanded for general intelligent robots. Conventional
robotic manipulation methods usually learn semantic representation of the
observation for action prediction, which ignores the scene-level spatiotemporal
dynamics for human goal completion. In this paper, we propose a dynamic
Gaussian Splatting method named ManiGaussian for multi-task robotic
manipulation, which mines scene dynamics via future scene reconstruction.
Specifically, we first formulate the dynamic Gaussian Splatting framework that
infers the semantics propagation in the Gaussian embedding space, where the
semantic representation is leveraged to predict the optimal robot action. Then,
we build a Gaussian world model to parameterize the distribution in our dynamic
Gaussian Splatting framework, which provides informative supervision in the
interactive environment via future scene reconstruction. We evaluate our
ManiGaussian on 10 RLBench tasks with 166 variations, and the results
demonstrate our framework can outperform the state-of-the-art methods by 13.1\%
in average success rate.

在这篇论文中，我们提出了一种名为 ManiGaussian 的动态高斯斑点方法，用于多任务机器人操作，该方法通过未来场景重建来挖掘场景动态性。我们首先制定了动态高斯斑点框架，推断高斯嵌入空间中的语义传播，利用语义表示来预测最佳机器人动作。然后，我们建立了一个高斯世界模型来参数化我们的动态高斯斑点框架中的分布，通过未来场景重建在交互环境中提供信息化监督。我们在 10 个 RLBench 任务的 166 个变体上评估了 ManiGaussian，并且结果表明我们的框架在平均成功率上比现有最先进方法提高了 13.1%。

ManiGaussian: 多任务机器人操作的动态高斯溅射

ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic  Manipulation

Recent works have shown that Large Language Models (LLMs) can be applied to
ground natural language to a wide variety of robot skills. However, in
practice, learning multi-task, language-conditioned robotic skills typically
requires large-scale data collection and frequent human intervention to reset
the environment or help correcting the current policies. In this work, we
propose a novel approach to efficiently learn general-purpose
language-conditioned robot skills from unstructured, offline and reset-free
data in the real world by exploiting a self-supervised visuo-lingual affordance
model, which requires annotating as little as 1% of the total data with
language. We evaluate our method in extensive experiments both in simulated and
real-world robotic tasks, achieving state-of-the-art performance on the
challenging CALVIN benchmark and learning over 25 distinct visuomotor
manipulation tasks with a single policy in the real world. We find that when
paired with LLMs to break down abstract natural language instructions into
subgoals via few-shot prompting, our method is capable of completing
long-horizon, multi-tier tasks in the real world, while requiring an order of
magnitude less data than previous approaches. Code and videos are available at
this http URL

本文提出了一种基于视觉语言感知模型的新方法，可以在真实世界中从非结构化、离线、无需重置的数据中高效地学习通用的、以语言为条件的机器人技能，实验表明该方法配合 LLMS 可以在真实世界中完成长期、多层次的任务，并比以往方法少使用一个数量级的数据。