Eye-in-hand cameras have shown promise in enabling greater sample efficiency
and generalization in vision-based robotic manipulation. However, for robotic
imitation, it is still expensive to have a human teleoperator collect large
amounts of expert demonstrations with a real robot. Videos of humans performing
tasks, on the other hand, are much cheaper to collect since they eliminate the
need for expertise in robotic teleoperation and can be quickly captured in a
wide range of scenarios. Therefore, human video demonstrations are a promising
data source for learning generalizable robotic manipulation policies at scale.
In this work, we augment narrow robotic imitation datasets with broad unlabeled
human video demonstrations to greatly enhance the generalization of eye-in-hand
visuomotor policies. Although a clear visual domain gap exists between human
and robot data, our framework does not need to employ any explicit domain
adaptation method, as we leverage the partial observability of eye-in-hand
cameras as well as a simple fixed image masking scheme. On a suite of eight
real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
improves the success rates of eye-in-hand manipulation policies by 58%
(absolute) on average, enabling robots to generalize to both new environment
configurations and new tasks that are unseen in the robot demonstration data.
See video results at this https URL .

本研究旨在利用标注不足的人类视频示范来改进基于视觉的机器人操作学习数据，通过引入图像遮蔽的方法，提高眼手相机机器人操作策略的成功率达 58％。

人机合作：通过眼手协同人类演示学习可推广的机器人操作

Giving Robots a Hand: Learning Generalizable Manipulation with  Eye-in-Hand Human Video Demonstrations

This paper presents a new method to describe spatio-temporal relations
between objects and hands, to recognize both interactions and activities within
video demonstrations of manual tasks. The approach exploits Scene Graphs to
extract key interaction features from image sequences, encoding at the same
time motion patterns and context. Additionally, the method introduces an
event-based automatic video segmentation and clustering, which allows to group
similar events, detecting also on the fly if a monitored activity is executed
correctly. The effectiveness of the approach was demonstrated in two
multi-subject experiments, showing the ability to recognize and cluster
hand-object and object-object interactions without prior knowledge of the
activity, as well as matching the same activity performed by different
subjects.

本文提出一种基于场景图，利用图像序列提取关键交互特征并编码动作模式和上下文的方法，同时引入基于事件的自动视频分割和聚类，成功地实现了识别手 - 物体和物体 - 物体交互，并匹配不同受试者执行的同一活动。

基于人类手动演示视频的自动交互和活动识别技术及其在异常检测中的应用

Automatic Interaction and Activity Recognition from Videos of Human  Manual Demonstrations with Application to Anomaly Detection

The ability to use inductive reasoning to extract general rules from multiple
observations is a vital indicator of intelligence. As humans, we use this
ability to not only interpret the world around us, but also to predict the
outcomes of the various interactions we experience. Generalising over multiple
observations is a task that has historically presented difficulties for
machines to grasp, especially when requiring computer vision. In this paper, we
propose a model that can extract general rules from video demonstrations by
simultaneously performing summarisation and translation. Our approach differs
from prior works by framing the problem as a multi-sequence-to-sequence task,
wherein summarisation is learnt by the model. This allows our model to utilise
edge cases that would otherwise be suppressed or discarded by traditional
summarisation techniques. Additionally, we show that our approach can handle
noisy specifications without the need for additional filtering methods. We
evaluate our model by synthesising programs from video demonstrations in the
Vizdoom environment achieving state-of-the-art results with a relative increase
of 11.75% program accuracy on prior works

本文介绍一种可以从视频演示中提取概括性规则的模型，并探讨了其处理多个序列的能力，从而相较于传统的概括技术可以更好地利用边缘情况，并无需进行额外的过滤。通过在 Vizdoom 环境中合成程序的实验，取得了相对于现有工作的 11.75% 的程序准确性的提升。

多樣視頻演示生成程序

Program Generation from Diverse Video Demonstrations

The imitation learning research community has recently made significant
progress towards the goal of enabling artificial agents to imitate behaviors
from video demonstrations alone. However, current state-of-the-art approaches
developed for this problem exhibit high sample complexity due, in part, to the
high-dimensional nature of video observations. Towards addressing this issue,
we introduce here a new algorithm called Visual Generative Adversarial
Imitation from Observation using a State Observer VGAIfO-SO. At its core,
VGAIfO-SO seeks to address sample inefficiency using a novel, self-supervised
state observer, which provides estimates of lower-dimensional proprioceptive
state representations from high-dimensional images. We show experimentally in
several continuous control environments that VGAIfO-SO is more sample efficient
than other IfO algorithms at learning from video-only demonstrations and can
sometimes even achieve performance close to the Generative Adversarial
Imitation from Observation (GAIfO) algorithm that has privileged access to the
demonstrator's proprioceptive state information.

介绍了一种名为 Visual Generative Adversarial Imitation from Observation using a State Observer (VGAIfO-SO) 的新算法，该算法使用自我监督的状态观察器从高维图像中提供低维本体感知状态表示的估计，从而更有效地从仅视频演示中学习，并且有时可以实现接近于有特权访问演示者本体感知状态信息的 GAIfO 算法的性能。