Given a natural language instruction and an input scene, our goal is to train
a model to output a manipulation program that can be executed by the robot.
Prior approaches for this task possess one of the following limitations: (i)
rely on hand-coded symbols for concepts limiting generalization beyond those
seen during training [1] (ii) infer action sequences from instructions but
require dense sub-goal supervision [2] or (iii) lack semantics required for
deeper object-centric reasoning inherent in interpreting complex instructions
[3]. In contrast, our approach can handle linguistic as well as perceptual
variations, end-to-end trainable and requires no intermediate supervision. The
proposed model uses symbolic reasoning constructs that operate on a latent
neural object-centric representation, allowing for deeper reasoning over the
input scene. Central to our approach is a modular structure consisting of a
hierarchical instruction parser and an action simulator to learn disentangled
action representations. Our experiments on a simulated environment with a 7-DOF
manipulator, consisting of instructions with varying number of steps and scenes
with different number of objects, demonstrate that our model is robust to such
variations and significantly outperforms baselines, particularly in the
generalization settings. The code, dataset and experiment videos are available
at this https URL

通过一种模块化结构，使用符号推理构造深度物体中心推理模型，从而训练出一种可以执行机器人操作的模型，此模型具有优秀的通用性和端到端的可训练性。

学习神经符号程序用于语言引导机器人操控

Learning Neuro-symbolic Programs for Language Guided Robot Manipulation

We propose a self-supervised capsule architecture for 3D point clouds. We
compute capsule decompositions of objects through permutation-equivariant
attention, and self-supervise the process by training with pairs of randomly
rotated objects. Our key idea is to aggregate the attention masks into semantic
keypoints, and use these to supervise a decomposition that satisfies the
capsule invariance/equivariance properties. This not only enables the training
of a semantically consistent decomposition, but also allows us to learn a
canonicalization operation that enables object-centric reasoning. To train our
neural network we require neither classification labels nor manually-aligned
training datasets. Yet, by learning an object-centric representation in a
self-supervised manner, our method outperforms the state-of-the-art on 3D point
cloud reconstruction, canonicalization, and unsupervised classification.

本研究提出了一种自监督胶囊体系结构用于 3D 点云，其通过置换等变性注意力计算对象的胶囊分解，并通过随机旋转对象对进行训练。我们的关键思想是将注意力掩码聚合成语义关键点，并使用这些来监督满足胶囊不变性 / 等变性属性的分解，从而使训练具有语义一致的分解，同时还可以学习一种使对象集中化推理的标准化操作。通过自监督学习对象集中的表示，我们的方法在 3D 点云重建，规范化和无监督分类方面优于最先进的方法。

规范姿态下的自监督胶囊

Canonical Capsules: Self-Supervised Capsules in Canonical Pose

Reasoning about objects, relations, and physics is central to human
intelligence, and a key goal of artificial intelligence. Here we introduce the
interaction network, a model which can reason about how objects in complex
systems interact, supporting dynamical predictions, as well as inferences about
the abstract properties of the system. Our model takes graphs as input,
performs object- and relation-centric reasoning in a way that is analogous to a
simulation, and is implemented using deep neural networks. We evaluate its
ability to reason about several challenging physical domains: n-body problems,
rigid-body collision, and non-rigid dynamics. Our results show it can be
trained to accurately simulate the physical trajectories of dozens of objects
over thousands of time steps, estimate abstract quantities such as energy, and
generalize automatically to systems with different numbers and configurations
of objects and relations. Our interaction network implementation is the first
general-purpose, learnable physics engine, and a powerful general framework for
reasoning about object and relations in a wide variety of complex real-world
domains.

本文介绍了交互网络模型，它能够推断复杂系统中物体之间相互作用的方式，支持对系统动态预测，以及对系统抽象属性的推理。该模型使用深度神经网络进行对象和关系居中的推理，并可以推理多种具有挑战性的物理领域。 报告结果表明，它能够训练准确地模拟数十个对象在数千个时间步骤上的物理轨迹，估计能量等抽象量，自动推广到具有不同数量和配置的对象和关系的系统中。 该交互网络的实现是第一个通用的，可学习的物理引擎，是处理各种复杂真实世界领域中物体和关系的强大的通用框架。