Recognizing human actions is a vital task for a humanoid robot, especially in
domains like programming by demonstration. Previous approaches on action
recognition primarily focused on the overall prevalent action being executed,
but we argue that bimanual human motion cannot always be described sufficiently
with a single action label. We present a system for frame-wise action
classification and segmentation in bimanual human demonstrations. The system
extracts symbolic spatial object relations from raw RGB-D video data captured
from the robot's point of view in order to build graph-based scene
representations. To learn object-action relations, a graph network classifier
is trained using these representations together with ground truth action labels
to predict the action executed by each hand.
We evaluated the proposed classifier on a new RGB-D video dataset showing
daily action sequences focusing on bimanual manipulation actions. It consists
of 6 subjects performing 9 tasks with 10 repetitions each, which leads to 540
video recordings with 2 hours and 18 minutes total playtime and per-hand ground
truth action labels for each frame. We show that the classifier is able to
reliably identify (action classification macro F1-score of 0.86) the true
executed action of each hand within its top 3 predictions on a frame-by-frame
basis without prior temporal action segmentation.

本文研究了一种针对双手人体动作的按帧分类和分割系统，通过从机器人视角获取的 RGB-D 视频数据提取符号空间物体关系来建立基于图的场景表示，并使用帧级的真实标签训练图网络分类器来预测每只手执行的动作，实验表明，这种分类器能够在不需要先前的时间动作分割的情况下，可靠地识别每只手的真实执行动作。