Human actions involving hand manipulations are structured according to the
making and breaking of hand-object contact, and human visual understanding of
action is reliant on anticipation of contact as is demonstrated by pioneering
work in cognitive science. Taking inspiration from this, we introduce
representations and models centered on contact, which we then use in action
prediction and anticipation. We annotate a subset of the EPIC Kitchens dataset
to include time-to-contact between hands and objects, as well as segmentations
of hands and objects. Using these annotations we train the Anticipation Module,
a module producing Contact Anticipation Maps and Next Active Object
Segmentations - novel low-level representations providing temporal and spatial
characteristics of anticipated near future action. On top of the Anticipation
Module we apply Egocentric Object Manipulation Graphs (Ego-OMG), a framework
for action anticipation and prediction. Ego-OMG models longer term temporal
semantic relations through the use of a graph modeling transitions between
contact delineated action states. Use of the Anticipation Module within Ego-OMG
produces state-of-the-art results, achieving 1st and 2nd place on the unseen
and seen test sets, respectively, of the EPIC Kitchens Action Anticipation
Challenge, and achieving state-of-the-art results on the tasks of action
anticipation and action prediction over EPIC Kitchens. We perform ablation
studies over characteristics of the Anticipation Module to evaluate their
utility.

本研究以手与物体的接触与断开作为行为结构的中心，从这个角度出发探究了人类视觉对行为的理解方式，并在此基础上提出了依据行为接触建模的方法，并为此设计了一套行为预测与研究框架，该框架的使用在第四次 EPIC Kitchens 行为预测挑战赛中达到了前沿水平。

通过第一人称视频中的接触表示预测行为

Forecasting Action through Contact Representations from First Person  Video

Grasping is natural for humans. However, it involves complex hand
configurations and soft tissue deformation that can result in complicated
regions of contact between the hand and the object. Understanding and modeling
this contact can potentially improve hand models, AR/VR experiences, and
robotic grasping. Yet, we currently lack datasets of hand-object contact paired
with other data modalities, which is crucial for developing and evaluating
contact modeling techniques. We introduce ContactPose, the first dataset of
hand-object contact paired with hand pose, object pose, and RGB-D images.
ContactPose has 2306 unique grasps of 25 household objects grasped with 2
functional intents by 50 participants, and more than 2.9 M RGB-D grasp images.
Analysis of ContactPose data reveals interesting relationships between hand
pose and contact. We use this data to rigorously evaluate various data
representations, heuristics from the literature, and learning methods for
contact modeling. Data, code, and trained models are available at
this https URL

介绍了 ContactPose 数据集，它是第一个把手对象接触配对其他数据模态的数据集，并用它来评估接触建模的各种数据表示、文献中的启发式方法和学习方法。

ContactPose：一份包含物体接触和手姿态的抓取数据集

ContactPose: A Dataset of Grasps with Object Contact and Hand Pose

Humans excel at grasping objects and manipulating them. Capturing human
grasps is important for understanding grasping behavior and reconstructing it
realistically in Virtual Reality (VR). However, grasp capture - capturing the
pose of a hand grasping an object, and orienting it w.r.t. the object - is
difficult because of the complexity and diversity of the human hand, and
occlusion. Reflective markers and magnetic trackers traditionally used to
mitigate this difficulty introduce undesirable artifacts in images and can
interfere with natural grasping behavior. We present preliminary work on a
completely marker-less algorithm for grasp capture from a video depicting a
grasp. We show how recent advances in 2D hand pose estimation can be used with
well-established optimization techniques. Uniquely, our algorithm can also
capture hand-object contact in detail and integrate it in the grasp capture
process. This is work in progress, find more details at this https URL
cc.gatech.edu/grasp_capture.html.

研究了通过使用最新的二维手姿势估计技术和优化技术进行的无标记算法来捕捉手抓握物体的过程，此算法也可以详细地捕捉手 - 物体接触并将其集成进抓握捕捉过程中。