To substantially enhance robot intelligence, there is a pressing need to
develop a large model that enables general-purpose robots to proficiently
undertake a broad spectrum of manipulation tasks, akin to the versatile
task-planning ability exhibited by LLMs. The vast diversity in objects, robots,
and manipulation tasks presents huge challenges. Our work introduces a
comprehensive framework to develop a foundation model for general robotic
manipulation that formalizes a manipulation task as contact synthesis.
Specifically, our model takes as input object and robot manipulator point
clouds, object physical attributes, target motions, and manipulation region
masks. It outputs contact points on the object and associated contact forces or
post-contact motions for robots to achieve the desired manipulation task. We
perform extensive experiments both in the simulation and real-world settings,
manipulating articulated rigid objects, rigid objects, and deformable objects
that vary in dimensionality, ranging from one-dimensional objects like ropes to
two-dimensional objects like cloth and extending to three-dimensional objects
such as plasticine. Our model achieves average success rates of around 90\%.
Supplementary materials and videos are available on our project website at
this https URL

为了极大增强机器人智能，我们提出了一个全面的框架来开发通用机器人操作的基础模型，并通过形式化操作任务为接触综合来实现。我们的模型以物体和机器人的点云、物体的物理属性、目标运动和操作区域掩码作为输入，并输出物体上的接触点以及机器人实现所需的接触力或接触后的运动。在仿真和实际世界环境中进行了广泛实验，操作的对象包括关节刚体、刚体和可变形物体，其维度范围从一维物体如绳索到二维物体如布料以及三维物体如软陶。我们的模型达到了约 90% 的平均成功率。

通用机器人接触合成中的 ManiFoundation 模型与任意物体和机器人的操作

ManiFoundation Model for General-Purpose Robotic Manipulation of Contact  Synthesis with Arbitrary Objects and Robots

The important manifestation of robot intelligence is the ability to naturally
interact and autonomously make decisions. Traditional approaches to robot
control often compartmentalize perception, planning, and decision-making,
simplifying system design but limiting the synergy between different
information streams. This compartmentalization poses challenges in achieving
seamless autonomous reasoning, decision-making, and action execution. To
address these limitations, a novel paradigm, named Vision-Language-Action tasks
for QUAdruped Robots (QUAR-VLA), has been introduced in this paper. This
approach tightly integrates visual information and instructions to generate
executable actions, effectively merging perception, planning, and
decision-making. The central idea is to elevate the overall intelligence of the
robot. Within this framework, a notable challenge lies in aligning fine-grained
instructions with visual perception information. This emphasizes the complexity
involved in ensuring that the robot accurately interprets and acts upon
detailed instructions in harmony with its visual observations. Consequently, we
propose QUAdruped Robotic Transformer (QUART), a family of VLA models to
integrate visual information and instructions from diverse modalities as input
and generates executable actions for real-world robots and present QUAdruped
Robot Dataset (QUARD), a large-scale multi-task dataset including navigation,
complex terrain locomotion, and whole-body manipulation tasks for training
QUART models. Our extensive evaluation (4000 evaluation trials) shows that our
approach leads to performant robotic policies and enables QUART to obtain a
range of emergent capabilities.

机器人智能的重要体现是自然互动和自主决策能力，本文引入了一种名为四足机器人视觉语言行为任务（QUAR-VLA）的新范式，通过紧密集成视觉信息和指令生成可执行动作来提高机器人的整体智能，使用 QUAdruped Robotic Transformer（QUART）系列模型将来自不同模态的视觉信息和指令作为输入，为真实世界的机器人生成可执行的动作，并提出了 QUAdruped Robot Dataset（QUARD）用于训练 QUART 模型，我们的广泛评估表明，我们的方法可以得到性能良好的机器人策略，并使 QUART 获得一系列新兴能力。

QUAR-VLA：四足机器人的视觉 - 语言 - 动作模型

QUAR-VLA: Vision-Language-Action Model for Quadruped Robots

Human body motions can be captured as a high-dimensional continuous signal
using motion sensor technologies. The resulting data can be surprisingly rich
in information, even when captured from persons with limited mobility. In this
work, we explore the use of limited upper-body motions, captured via motion
sensors, as inputs to control a 7 degree-of-freedom assistive robotic arm. It
is possible that even dense sensor signals lack the salient information and
independence necessary for reliable high-dimensional robot control. As the
human learns over time in the context of this limitation, intelligence on the
robot can be leveraged to better identify key learning challenges, provide
useful feedback, and support individuals until the challenges are managed. In
this short paper, we examine two uninjured participants' data from an ongoing
study, to extract preliminary results and share insights. We observe
opportunities for robot intelligence to step in, including the identification
of inconsistencies in time spent across all control dimensions, asymmetries in
individual control dimensions, and user progress in learning. Machine reasoning
about these situations may facilitate novel interface learning in the future.

本文研究采用运动传感器技术对有限活动能力的人体上半身动作进行捕捉，将其作为输入来控制 7 自由度辅助机器臂。通过对两位参与者的实验数据进行初步分析，观察到利用机器智能可以发现高维机器人控制中一些关键的难点和不一致性，从而促进未来接口学习的发展。