To build general robotic agents that can operate in many environments, it is
often imperative for the robot to collect experience in the real world.
However, this is often not feasible due to safety, time, and hardware
restrictions. We thus propose leveraging the next best thing as real-world
experience: internet videos of humans using their hands. Visual priors, such as
visual features, are often learned from videos, but we believe that more
information from videos can be utilized as a stronger prior. We build a
learning algorithm, VideoDex, that leverages visual, action, and physical
priors from human video datasets to guide robot behavior. These actions and
physical priors in the neural network dictate the typical human behavior for a
particular robot task. We test our approach on a robot arm and dexterous
hand-based system and show strong results on various manipulation tasks,
outperforming various state-of-the-art methods. Videos at
this https URL

利用网络视频的人类手部动作行为作为真实世界经验来指导机器人任务，使用人类视频数据集的视觉、动作、物理规律作为机器人行为的先验知识的学习算法 VideoDex，在机器人手臂和灵巧的基于手的系统上进行测试，并在各种操作任务上显示出强大的结果。

从互联网视频中学习手眼协调技能

VideoDex: Learning Dexterity from Internet Videos

Recent advances in image-based human pose estimation make it possible to
capture 3D human motion from a single RGB video. However, the inherent depth
ambiguity and self-occlusion in a single view prohibit the recovery of as
high-quality motion as multi-view reconstruction. While multi-view videos are
not common, the videos of a celebrity performing a specific action are usually
abundant on the Internet. Even if these videos were recorded at different time
instances, they would encode the same motion characteristics of the person.
Therefore, we propose to capture human motion by jointly analyzing these
Internet videos instead of using single videos separately. However, this new
task poses many new challenges that cannot be addressed by existing methods, as
the videos are unsynchronized, the camera viewpoints are unknown, the
background scenes are different, and the human motions are not exactly the same
among videos. To address these challenges, we propose a novel
optimization-based framework and experimentally demonstrate its ability to
recover much more precise and detailed motion from multiple videos, compared
against monocular motion capture methods.

该研究提出了一种基于优化框架的方法，可以从互联网视频中共同分析人类动作，以替代传统单一视角的运动捕捉方法并提高精度。

从互联网视频中获取动作捕捉

Motion Capture from Internet Videos

Hands are the central means by which humans manipulate their world and being
able to reliably extract hand state information from Internet videos of humans
engaged in their hands has the potential to pave the way to systems that can
learn from petabytes of video data. This paper proposes steps towards this by
inferring a rich representation of hands engaged in interaction method that
includes: hand location, side, contact state, and a box around the object in
contact. To support this effort, we gather a large-scale dataset of hands in
contact with objects consisting of 131 days of footage as well as a 100K
annotated hand-contact video frame dataset. The learned model on this dataset
can serve as a foundation for hand-contact understanding in videos. We
quantitatively evaluate it both on its own and in service of predicting and
learning from 3D meshes of human hands.

本文提出了一种从互联网视频中提取手部状态信息的方法，通过收集大规模数据集并建立学习模型，该方法能够推测出手部位置、接触状态以及所接触的物体，支持 3D 网格的建模。

互联网规模下理解人类接触的双手

Understanding Human Hands in Contact at Internet Scale

This paper presents a video summarization technique for an Internet video to
provide a quick way to overview its content. This is a challenging problem
because finding important or informative parts of the original video requires
to understand its content. Furthermore the content of Internet videos is very
diverse, ranging from home videos to documentaries, which makes video
summarization much more tough as prior knowledge is almost not available. To
tackle this problem, we propose to use deep video features that can encode
various levels of content semantics, including objects, actions, and scenes,
improving the efficiency of standard video summarization techniques. For this,
we design a deep neural network that maps videos as well as descriptions to a
common semantic space and jointly trained it with associated pairs of videos
and descriptions. To generate a video summary, we extract the deep features
from each segment of the original video and apply a clustering-based
summarization technique to them. We evaluate our video summaries using the
SumMe dataset as well as baseline approaches. The results demonstrated the
advantages of incorporating our deep semantic features in a video summarization
technique.

该论文提出了一种基于深度视频特征和聚类的视频摘要技术，旨在更高效地生成互联网视频的概览，该技术通过设计一种映射视频和描述的深度神经网络来将视频元素转化为语义空间。