Large Language Models (LLMs) and strong vision models have enabled rapid
research and development in the field of Vision-Language-Action models that
enable robotic control. The main objective of these methods is to develop a
generalist policy that can control robots with various embodiments. However, in
industrial robotic applications such as automated assembly and disassembly,
some tasks, such as insertion, demand greater accuracy and involve intricate
factors like contact engagement, friction handling, and refined motor skills.
Implementing these skills using a generalist policy is challenging because
these policies might integrate further sensory data, including force or torque
measurements, for enhanced precision. In our method, we present a global
control policy based on LLMs that can transfer the control policy to a finite
set of skills that are specifically trained to perform high-precision tasks
through dynamic context switching. The integration of LLMs into this framework
underscores their significance in not only interpreting and processing language
inputs but also in enriching the control mechanisms for diverse and intricate
robotic operations.

基于大型语言模型的全局控制策略结合了语言输入的解释和处理能力，为多样化和复杂的机器人操作丰富了控制机制。

基于自然语言的装配技术发展

Towards Natural Language-Driven Assembly Using Foundation Models

Large policies pretrained on a combination of Internet-scale vision-language
data and diverse robot demonstrations have the potential to change how we teach
robots new skills: rather than training new behaviors from scratch, we can
fine-tune such vision-language-action (VLA) models to obtain robust,
generalizable policies for visuomotor control. Yet, widespread adoption of VLAs
for robotics has been challenging as 1) existing VLAs are largely closed and
inaccessible to the public, and 2) prior work fails to explore methods for
efficiently fine-tuning VLAs for new tasks, a key component for adoption.
Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source
VLA trained on a diverse collection of 970k real-world robot demonstrations.
OpenVLA builds on a Llama 2 language model combined with a visual encoder that
fuses pretrained features from DINOv2 and SigLIP. As a product of the added
data diversity and new model components, OpenVLA demonstrates strong results
for generalist manipulation, outperforming closed models such as RT-2-X (55B)
by 16.5% in absolute task success rate across 29 tasks and multiple robot
embodiments, with 7x fewer parameters. We further show that we can effectively
fine-tune OpenVLA for new settings, with especially strong generalization
results in multi-task environments involving multiple objects and strong
language grounding abilities, and outperform expressive from-scratch imitation
learning methods such as Diffusion Policy by 20.4%. We also explore compute
efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned
on consumer GPUs via modern low-rank adaptation methods and served efficiently
via quantization without a hit to downstream success rate. Finally, we release
model checkpoints, fine-tuning notebooks, and our PyTorch codebase with
built-in support for training VLAs at scale on Open X-Embodiment datasets.

通过使用互联网规模的视觉 - 语言数据和多样化的机器人演示进行预训练的大型模型，可以改变我们教授机器人新技能的方式：相对于从零开始训练新行为，我们可以对这种视觉 - 语言 - 动作（VLA）模型进行微调，以获得稳健、具有广泛适用性的视觉 - 动作策略。我们介绍了 OpenVLA，这是一个 7B 参数的开源 VLA，它在多样化的 970k 真实世界机器人演示数据集上进行了训练。OpenVLA 基于 Llama 2 语言模型和可视编码器，融合了 DINOv2 和 SigLIP 的预训练特征。通过增加数据多样性和新的模型组件，OpenVLA 在广义操纵方面表现出色，在 29 个任务和多个机器人实体上的绝对任务成功率上优于 RT-2-X（55B）等闭源模型 16.5%，并且参数数量仅为其七分之一。我们进一步展示了 OpenVLA 对于新环境的有效微调，在涉及多个对象和强大语言基础能力的多任务环境中，具有特别强大的泛化能力，且优于 Diffusion Policy 等基于从头开始的模仿学习方法 20.4%。此外，我们还探索了计算效率；作为额外贡献，我们表明 OpenVLA 可以通过现代低秩适应方法在消费级 GPU 上进行微调，并通过量化方式进行有效服务，而不会影响下游的成功率。最后，我们发布了模型检查点、微调笔记本以及基于 PyTorch 的代码库，支持在 Open X-Embodiment 数据集上规模化训练 VLA。

OpenVLA：一个开放源代码的视觉语言行动模型

OpenVLA: An Open-Source Vision-Language-Action Model

Deep learning has demonstrated remarkable success across many domains,
including computer vision, natural language processing, and reinforcement
learning. Representative artificial neural networks in these fields span
convolutional neural networks, Transformers, and deep Q-networks. Built upon
unimodal neural networks, numerous multi-modal models have been introduced to
address a range of tasks such as visual question answering, image captioning,
and speech recognition. The rise of instruction-following robotic policies in
embodied AI has spurred the development of a novel category of multi-modal
models known as vision-language-action models (VLAs). Their multi-modality
capability has become a foundational element in robot learning. Various methods
have been proposed to enhance traits such as versatility, dexterity, and
generalizability. Some models focus on refining specific components through
pretraining. Others aim to develop control policies adept at predicting
low-level actions. Certain VLAs serve as high-level task planners capable of
decomposing long-horizon tasks into executable subtasks. Over the past few
years, a myriad of VLAs have emerged, reflecting the rapid advancement of
embodied AI. Therefore, it is imperative to capture the evolving landscape
through a comprehensive survey.

综合调查了深度学习、多模态模型、视觉 - 语言 - 动作模型、具身人工智能的快速发展。

视觉语言行为模型在具身人工智能中的调查

A Survey on Vision-Language-Action Models for Embodied AI

Recent vision-language-action (VLA) models rely on 2D inputs, lacking
integration with the broader realm of the 3D physical world. Furthermore, they
perform action prediction by learning a direct mapping from perception to
action, neglecting the vast dynamics of the world and the relations between
actions and dynamics. In contrast, human beings are endowed with world models
that depict imagination about future scenarios to plan actions accordingly. To
this end, we propose 3D-VLA by introducing a new family of embodied foundation
models that seamlessly link 3D perception, reasoning, and action through a
generative world model. Specifically, 3D-VLA is built on top of a 3D-based
large language model (LLM), and a set of interaction tokens is introduced to
engage with the embodied environment. Furthermore, to inject generation
abilities into the model, we train a series of embodied diffusion models and
align them into the LLM for predicting the goal images and point clouds. To
train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
extracting vast 3D-related information from existing robotics datasets. Our
experiments on held-in datasets demonstrate that 3D-VLA significantly improves
the reasoning, multimodal generation, and planning capabilities in embodied
environments, showcasing its potential in real-world applications.

提出了一种基于 3D 感知、推理和行动的生成世界模型的 3D-VLA 模型，通过引入一系列交互令牌与具体环境进行交互，训练一系列融入大规模 3D 语言模型的生成扩散模型以预测目标图像和点云，并在大规模数据集上的实验中展示了 3D-VLA 在推理、多模态生成和规划能力上的显著改进，展示了其在真实世界应用中的潜力。

3D-VLA：一个基于三维视觉 - 语言 - 动作的生成式世界模型

3D-VLA: A 3D Vision-Language-Action Generative World Model

We study how vision-language models trained on Internet-scale data can be
incorporated directly into end-to-end robotic control to boost generalization
and enable emergent semantic reasoning. Our goal is to enable a single
end-to-end trained model to both learn to map robot observations to actions and
enjoy the benefits of large-scale pretraining on language and vision-language
data from the web. To this end, we propose to co-fine-tune state-of-the-art
vision-language models on both robotic trajectory data and Internet-scale
vision-language tasks, such as visual question answering. In contrast to other
approaches, we propose a simple, general recipe to achieve this goal: in order
to fit both natural language responses and robotic actions into the same
format, we express the actions as text tokens and incorporate them directly
into the training set of the model in the same way as natural language tokens.
We refer to such category of models as vision-language-action models (VLA) and
instantiate an example of such a model, which we call RT-2. Our extensive
evaluation (6k evaluation trials) shows that our approach leads to performant
robotic policies and enables RT-2 to obtain a range of emergent capabilities
from Internet-scale training. This includes significantly improved
generalization to novel objects, the ability to interpret commands not present
in the robot training data (such as placing an object onto a particular number
or icon), and the ability to perform rudimentary reasoning in response to user
commands (such as picking up the smallest or largest object, or the one closest
to another object). We further show that incorporating chain of thought
reasoning allows RT-2 to perform multi-stage semantic reasoning, for example
figuring out which object to pick up for use as an improvised hammer (a rock),
or which type of drink is best suited for someone who is tired (an energy
drink).

通过在互联网规模的数据上训练视觉语言模型，将其直接融入端到端的机器人控制中，提高泛化能力和实现新兴的语义推理。我们提出了一种简单通用的方法，在模型训练集中直接将行动表现为文本标记，将自然语言回答和机器人行动合并到同一个格式中，从而实现了单一端到端训练模型的目标。我们将这类模型称为视觉语言行动模型（VLA），并以 RT-2 为例进行了实例化。广泛的评估结果表明，我们的方法可以得到性能优越的机器人策略，并使 RT-2 在互联网规模的训练中获得一系列新兴能力。这包括对新对象的显著改进的泛化能力，解释不在机器人训练数据中的命令（比如将物体放在特定的编号或图标上）的能力，以及对用户指令做出初步推理的能力（比如选择最小或最大的物体，或者离另一个物体最近的物体）。我们进一步说明了通过思维链式推理，RT-2 可以进行多阶段的语义推理，例如找到作为非正式锤子使用的物体（一块石头），或者适合疲劳人群的饮料类型（一种能量饮料）。