We study how the choice of visual perspective affects learning and
generalization in the context of physical manipulation from raw sensor
observations. Compared with the more commonly used global third-person
perspective, a hand-centric (eye-in-hand) perspective affords reduced
observability, but we find that it consistently improves training efficiency
and out-of-distribution generalization. These benefits hold across a variety of
learning algorithms, experimental settings, and distribution shifts, and for
both simulated and real robot apparatuses. However, this is only the case when
hand-centric observability is sufficient; otherwise, including a third-person
perspective is necessary for learning, but also harms out-of-distribution
generalization. To mitigate this, we propose to regularize the third-person
information stream via a variational information bottleneck. On six
representative manipulation tasks with varying hand-centric observability
adapted from the Meta-World benchmark, this results in a state-of-the-art
reinforcement learning agent operating from both perspectives improving its
out-of-distribution generalization on every task. While some practitioners have
long put cameras in the hands of robots, our work systematically analyzes the
benefits of doing so and provides simple and broadly applicable insights for
improving end-to-end learned vision-based robotic manipulation.

本研究发现相较于全局第三人称视角，手心视角可以提高物理操作的训练效率和泛化能力，并适用于各种算法、实验环境和分布转变。

基于视觉的机械臂同时需要从其手部感知视觉

Vision-Based Manipulators Need to Also See from Their Hands

Recent work has documented the susceptibility of deep learning systems to
adversarial examples, but most such attacks directly manipulate the digital
input to a classifier. Although a smaller line of work considers physical
adversarial attacks, in all cases these involve manipulating the object of
interest, e.g., putting a physical sticker on an object to misclassify it, or
manufacturing an object specifically intended to be misclassified. In this
work, we consider an alternative question: is it possible to fool deep
classifiers, over all perceived objects of a certain type, by physically
manipulating the camera itself? We show that by placing a carefully crafted and
mainly-translucent sticker over the lens of a camera, one can create universal
perturbations of the observed images that are inconspicuous, yet misclassify
target objects as a different (targeted) class. To accomplish this, we propose
an iterative procedure for both updating the attack perturbation (to make it
adversarial for a given classifier), and the threat model itself (to ensure it
is physically realizable). For example, we show that we can achieve
physically-realizable attacks that fool ImageNet classifiers in a targeted
fashion 49.6% of the time. This presents a new class of physically-realizable
threat models to consider in the context of adversarially robust machine
learning. Our demo video can be viewed at: this https URL

本文主要研究了利用对摄像头物理操纵的方式实施对深度神经网络的物理对抗攻击，并提出了一种迭代的攻击方法，可以使攻击不易被察觉，实现了针对 ImageNet 分类器的 49.6% 的攻击成功率。