Acquiring a multi-task imitation policy in 3D manipulation poses challenges
in terms of scene understanding and action prediction. Current methods employ
both 3D representation and multi-view 2D representation to predict the poses of
the robot's end-effector. However, they still require a considerable amount of
high-quality robot trajectories, and suffer from limited generalization in
unseen tasks and inefficient execution in long-horizon reasoning. In this
paper, we propose SAM-E, a novel architecture for robot manipulation by
leveraging a vision-foundation model for generalizable scene understanding and
sequence imitation for long-term action reasoning. Specifically, we adopt
Segment Anything (SAM) pre-trained on a huge number of images and promptable
masks as the foundation model for extracting task-relevant features, and employ
parameter-efficient fine-tuning on robot data for a better understanding of
embodied scenarios. To address long-horizon reasoning, we develop a novel
multi-channel heatmap that enables the prediction of the action sequence in a
single pass, notably enhancing execution efficiency. Experimental results from
various instruction-following tasks demonstrate that SAM-E achieves superior
performance with higher execution efficiency compared to the baselines, and
also significantly improves generalization in few-shot adaptation to new tasks.

通过 SAM-E，我们提出了一种新颖的机器人操作体系结构，通过视觉基础模型进行可 generalizable 的场景理解和长期 action 推理的序列模仿，在各种指令跟踪任务中展现出卓越的性能和更高的执行效率，并显著提高了对新任务的少样本适应性。

SAM-E：利用序列模仿与视觉基础模型实现身体操作

SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for  Embodied Manipulation

The convergence of embodied agents and large language models (LLMs) has
brought significant advancements to embodied instruction following.
Particularly, the strong reasoning capabilities of LLMs make it possible for
robots to perform long-horizon tasks without expensive annotated
demonstrations. However, public benchmarks for testing the long-horizon
reasoning capabilities of language-conditioned robots in various scenarios are
still missing. To fill this gap, this work focuses on the tabletop manipulation
task and releases a simulation benchmark, \textit{LoHoRavens}, which covers
various long-horizon reasoning aspects spanning color, size, space, arithmetics
and reference. Furthermore, there is a key modality bridging problem for
long-horizon manipulation tasks with LLMs: how to incorporate the observation
feedback during robot execution for the LLM's closed-loop planning, which is
however less studied by prior work. We investigate two methods of bridging the
modality gap: caption generation and learnable interface for incorporating
explicit and implicit observation feedback to the LLM, respectively. These
methods serve as the two baselines for our proposed benchmark. Experiments show
that both methods struggle to solve some tasks, indicating long-horizon
manipulation tasks are still challenging for current popular models. We expect
the proposed public benchmark and baselines can help the community develop
better models for long-horizon tabletop manipulation tasks.

本研究基于体验工具人和大型语言模型的融合，在长时序任务中通过引入颜色、尺寸、空间、算术和参考等多个方面的推理来进行桌面操作，提出了一个模拟基准测试过程，同时探索通过标题生成和可学习接口来弥合模态差距，以改进当前流行模型在长时序桌面操作任务中的表现。