Enabling home-assistant robots to perceive and manipulate a diverse range of
3D objects based on human language instructions is a pivotal challenge. Prior
research has predominantly focused on simplistic and task-oriented
instructions, i.e., "Slide the top drawer open". However, many real-world tasks
demand intricate multi-step reasoning, and without human instructions, these
will become extremely difficult for robot manipulation. To address these
challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15
distinct manipulation tasks, containing over 4500 episodes meticulously
annotated with fine-grained language instructions. We split the long-term task
process into several steps, with each step having a natural language
instruction. Moreover, we propose a novel learning framework that completes the
manipulation task step-by-step according to the fine-grained instructions.
Specifically, we first identify the instruction to execute, taking into account
visual observations and the end-effector's current state. Subsequently, our
approach facilitates explicit learning through action-prompts and
perception-prompts to promote manipulation-aware cross-modality alignment.
Leveraging both visual observations and linguistic guidance, our model outputs
a sequence of actionable predictions for manipulation, including contact points
and end-effector poses. We evaluate our method and baselines using the proposed
benchmark NrVLM. The experimental results demonstrate the effectiveness of our
approach. For additional details, please refer to
this https URL

通过提出一个包含 15 个不同操纵任务的综合基准测试，该论文引入了一个以人类语言指令为基础的家庭助理机器人感知和操作多种 3D 物体的重大挑战，并通过建立一个新的学习框架逐步完成操纵任务。

自然 VLM：利用细粒度自然语言进行支配引导的视觉操作

NaturalVLM: Leveraging Fine-grained Natural Language for  Affordance-Guided Visual Manipulation

Perceiving and manipulating 3D articulated objects in diverse environments is
essential for home-assistant robots. Recent studies have shown that point-level
affordance provides actionable priors for downstream manipulation tasks.
However, existing works primarily focus on single-object scenarios with
homogeneous agents, overlooking the realistic constraints imposed by the
environment and the agent's morphology, e.g., occlusions and physical
limitations. In this paper, we propose an environment-aware affordance
framework that incorporates both object-level actionable priors and environment
constraints. Unlike object-centric affordance approaches, learning
environment-aware affordance faces the challenge of combinatorial explosion due
to the complexity of various occlusions, characterized by their quantities,
geometries, positions and poses. To address this and enhance data efficiency,
we introduce a novel contrastive affordance learning framework capable of
training on scenes containing a single occluder and generalizing to scenes with
complex occluder combinations. Experiments demonstrate the effectiveness of our
proposed approach in learning affordance considering environment constraints.

通过结合物体层面的可行先验和环境约束，我们提出了一个环境感知的可行性框架，该框架能够在考虑环境约束的情况下学习可行性，对于包含单个遮挡物和复杂遮挡物组合的场景具有良好的泛化效果。