Enabling home-assistant robots to perceive and manipulate a diverse range of
3D objects based on human language instructions is a pivotal challenge. Prior
research has predominantly focused on simplistic and task-oriented
instructions, i.e., "Slide the top drawer open". However, many real-world tasks
demand intricate multi-step reasoning, and without human instructions, these
will become extremely difficult for robot manipulation. To address these
challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15
distinct manipulation tasks, containing over 4500 episodes meticulously
annotated with fine-grained language instructions. We split the long-term task
process into several steps, with each step having a natural language
instruction. Moreover, we propose a novel learning framework that completes the
manipulation task step-by-step according to the fine-grained instructions.
Specifically, we first identify the instruction to execute, taking into account
visual observations and the end-effector's current state. Subsequently, our
approach facilitates explicit learning through action-prompts and
perception-prompts to promote manipulation-aware cross-modality alignment.
Leveraging both visual observations and linguistic guidance, our model outputs
a sequence of actionable predictions for manipulation, including contact points
and end-effector poses. We evaluate our method and baselines using the proposed
benchmark NrVLM. The experimental results demonstrate the effectiveness of our
approach. For additional details, please refer to
this https URL

通过提出一个包含 15 个不同操纵任务的综合基准测试，该论文引入了一个以人类语言指令为基础的家庭助理机器人感知和操作多种 3D 物体的重大挑战，并通过建立一个新的学习框架逐步完成操纵任务。

自然 VLM：利用细粒度自然语言进行支配引导的视觉操作

NaturalVLM: Leveraging Fine-grained Natural Language for  Affordance-Guided Visual Manipulation

We present an end-to-end diffusion-based method for editing videos with human
language instructions, namely $\textbf{InstructVid2Vid}$. Our approach enables
the editing of input videos based on natural language instructions without any
per-example fine-tuning or inversion. The proposed InstructVid2Vid model
combines a pretrained image generation model, Stable Diffusion, with a
conditional 3D U-Net architecture to generate time-dependent sequence of video
frames. To obtain the training data, we incorporate the knowledge and expertise
of different models, including ChatGPT, BLIP, and Tune-a-Video, to synthesize
video-instruction triplets, which is a more cost-efficient alternative to
collecting data in real-world scenarios. To improve the consistency between
adjacent frames of generated videos, we propose the Frame Difference Loss,
which is incorporated during the training process. During inference, we extend
the classifier-free guidance to text-video input to guide the generated
results, making them more related to both the input video and instruction.
Experiments demonstrate that InstructVid2Vid is able to generate high-quality,
temporally coherent videos and perform diverse edits, including attribute
editing, change of background, and style transfer. These results highlight the
versatility and effectiveness of our proposed method. Code is released in
$\href{this https URL}{InstructVid2Vid}$.

本文介绍了一种基于预训练模型和条件式 3D U-Net 结构的、可以通过自然语言指令编辑视频的方法，同时提出了一种新的视帧差异损失函数，可以在训练过程中提高生成视频的时序一致性，实验表明该方法生成的视频质量高、时序连贯，能够进行多种视频编辑操作。