Action recognition from an egocentric viewpoint is a crucial perception task
in robotics and enables a wide range of human-robot interactions. While most
computer vision approaches prioritize the RGB camera, the Depth modality -
which can further amplify the subtleties of actions from an egocentric
perspective - remains underexplored. Our work focuses on recognizing actions
from egocentric RGB and Depth modalities in an industry-like environment. To
study this problem, we consider the recent MECCANO dataset, which provides a
wide range of assembling actions. Our framework is based on the 3D Video SWIN
Transformer to encode both RGB and Depth modalities effectively. To address the
inherent skewness in real-world multimodal action occurrences, we propose a
training strategy using an exponentially decaying variant of the focal loss
modulating factor. Additionally, to leverage the information in both RGB and
Depth modalities, we opt for late fusion to combine the predictions from each
modality. We thoroughly evaluate our method on the action recognition task of
the MECCANO dataset, and it significantly outperforms the prior work. Notably,
our method also secured first place at the multimodal action recognition
challenge at ICIAP 2023.

我们的研究集中在认识行为，特别是来自以自我为中心视角的行为，并结合深度模态从工业环境中识别行为。针对现实世界中多模态行为发生的困难，我们提出了一种训练策略，并通过后期融合来结合来自每种模态的预测，这在 MECCANO 数据集上明显优于先前的工作，并在 ICIAP 2023 的多模态行为识别挑战中获得第一名。

工业场景中的主体视角 RGB + 深度行为识别

Egocentric RGB+Depth Action Recognition in Industry-Like Settings

We focus on the task of everyday hand pose estimation from egocentric
viewpoints. For this task, we show that depth sensors are particularly
informative for extracting near-field interactions of the camera wearer with
his/her environment. Despite the recent advances in full-body pose estimation
using Kinect-like sensors, reliable monocular hand pose estimation in RGB-D
images is still an unsolved problem. The problem is considerably exacerbated
when analyzing hands performing daily activities from a first-person viewpoint,
due to severe occlusions arising from object manipulations and a limited
field-of-view. Our system addresses these difficulties by exploiting strong
priors over viewpoint and pose in a discriminative tracking-by-detection
framework. Our priors are operationalized through a photorealistic synthetic
model of egocentric scenes, which is used to generate training data for
learning depth-based pose classifiers. We evaluate our approach on an annotated
dataset of real egocentric object manipulation scenes and compare to both
commercial and academic approaches. Our method provides state-of-the-art
performance for both hand detection and pose estimation in egocentric RGB-D
images.

本文提出一种基于深度传感器的、具有先验的手势识别系统，使用包含自我场景的光辉合成模型来生成训练数据，并在真实环境下进行测试。结果表明，该方法在单目 RGB-D 图像中的手部检测和姿态估计方面均具有最先进的性能。