We present NeuralLabeling, a labeling approach and toolset for annotating a
scene using either bounding boxes or meshes and generating segmentation masks,
affordance maps, 2D bounding boxes, 3D bounding boxes, 6DOF object poses, depth
maps and object meshes. NeuralLabeling uses Neural Radiance Fields (NeRF) as
renderer, allowing labeling to be performed using 3D spatial tools while
incorporating geometric clues such as occlusions, relying only on images
captured from multiple viewpoints as input. To demonstrate the applicability of
NeuralLabeling to a practical problem in robotics, we added ground truth depth
maps to 30000 frames of transparent object RGB and noisy depth maps of glasses
placed in a dishwasher captured using an RGBD sensor, yielding the
Dishwasher30k dataset. We show that training a simple deep neural network with
supervision using the annotated depth maps yields a higher reconstruction
performance than training with the previously applied weakly supervised
approach.

我们提出了 NeuralLabeling，这是一种使用边界框或网格进行场景标注的方法和工具集，可以生成分割掩膜、可用性地图、2D 边界框、3D 边界框、6DOF 物体姿态、深度图和物体网格。

神经标注：使用神经辐射场标注视觉数据的多用途工具集

NeuralLabeling: A versatile toolset for labeling vision datasets using  Neural Radiance Fields

Flexible, goal-directed behavior is a fundamental aspect of human life. Based
on the free energy minimization principle, the theory of active inference
formalizes the generation of such behavior from a computational neuroscience
perspective. Based on the theory, we introduce an output-probabilistic,
temporally predictive, modular artificial neural network architecture, which
processes sensorimotor information, infers behavior-relevant aspects of its
world, and invokes highly flexible, goal-directed behavior. We show that our
architecture, which is trained end-to-end to minimize an approximation of free
energy, develops latent states that can be interpreted as affordance maps. That
is, the emerging latent states signal which actions lead to which effects
dependent on the local context. In combination with active inference, we show
that flexible, goal-directed behavior can be invoked, incorporating the
emerging affordance maps. As a result, our simulated agent flexibly steers
through continuous spaces, avoids collisions with obstacles, and prefers
pathways that lead to the goal with high certainty. Additionally, we show that
the learned agent is highly suitable for zero-shot generalization across
environments: After training the agent in a handful of fixed environments with
obstacles and other terrains affecting its behavior, it performs similarly well
in procedurally generated environments containing different amounts of
obstacles and terrains of various sizes at different locations.

基于自由能最小化原则的主动推理理论，提出了一个输出 - 概率、时间预测、模块化人工神经网络结构，能够处理感觉运动信息，推理与行为相关的世界方面，并调用高度灵活、目标导向的行为，其中包括利用形成的地形图灵活驾驶、避免障碍和选择通往目的地的路径。另外，研究表明该学习代理适用于进行零样本推广，适用于不同地形的环境。

仿真智能体的可支配性推断和主动运动控制

Inference of Affordances and Active Motor Control in Simulated Agents

Does having visual priors (e.g. the ability to detect objects) facilitate
learning to perform vision-based manipulation (e.g. picking up objects)? We
study this problem under the framework of transfer learning, where the model is
first trained on a passive vision task, and adapted to perform an active
manipulation task. We find that pre-training on vision tasks significantly
improves generalization and sample efficiency for learning to manipulate
objects. However, realizing these gains requires careful selection of which
parts of the model to transfer. Our key insight is that outputs of standard
vision models highly correlate with affordance maps commonly used in
manipulation. Therefore, we explore directly transferring model parameters from
vision networks to affordance prediction networks, and show that this can
result in successful zero-shot adaptation, where a robot can pick up certain
objects with zero robotic experience. With just a small amount of robotic
experience, we can further fine-tune the affordance model to achieve better
results. With just 10 minutes of suction experience or 1 hour of grasping
experience, our method achieves ~80% success rate at picking up novel objects.

使用迁移学习框架，通过对视觉网络和物体操作网络进行预训练和微调，成功地实现了零经验机器人对物体的拾取任务，且只需少量的机器人经验即可获得更好的效果。