We introduce RotateIt, a system that enables fingertip-based object rotation
along multiple axes by leveraging multimodal sensory inputs. Our system is
trained in simulation, where it has access to ground-truth object shapes and
physical properties. Then we distill it to operate on realistic yet noisy
simulated visuotactile and proprioceptive sensory inputs. These multimodal
inputs are fused via a visuotactile transformer, enabling online inference of
object shapes and physical properties during deployment. We show significant
performance improvements over prior methods and the importance of visual and
tactile sensing.

我们介绍了 RotateIt，这是一个通过利用多模式感知输入实现多轴指尖物体旋转的系统。系统在模拟环境中训练，可以访问物体的真实形状和物理属性，然后通过模拟的视觉触觉和本体觉感知输入进行操作。通过触视觉变压器将多模态输入融合，实现了部署过程中对物体形状和物理属性进行在线推断，显示出较之前方法的显著性能改进以及视觉和触觉感知的重要性。

视觉和触觉的手持物体总体旋转

General In-Hand Object Rotation with Vision and Touch

Learning to navigate in complex environments with dynamic elements is an
important milestone in developing AI agents. In this work we formulate the
navigation question as a reinforcement learning problem and show that data
efficiency and task performance can be dramatically improved by relying on
additional auxiliary tasks leveraging multimodal sensory inputs. In particular
we consider jointly learning the goal-driven reinforcement learning problem
with auxiliary depth prediction and loop closure classification tasks. This
approach can learn to navigate from raw sensory input in complicated 3D mazes,
approaching human-level performance even under conditions where the goal
location changes frequently. We provide detailed analysis of the agent
behaviour, its ability to localise, and its network activity dynamics, showing
that the agent implicitly learns key navigation abilities.

利用增强学习问题和多模态感应输入的辅助深度预测和循环闭环分类任务，可以学习从复杂 3D 迷宫中进行导航并接近人类级别表现