Action recognition from an egocentric viewpoint is a crucial perception task
in robotics and enables a wide range of human-robot interactions. While most
computer vision approaches prioritize the RGB camera, the Depth modality -
which can further amplify the subtleties of actions from an egocentric
perspective - remains underexplored. Our work focuses on recognizing actions
from egocentric RGB and Depth modalities in an industry-like environment. To
study this problem, we consider the recent MECCANO dataset, which provides a
wide range of assembling actions. Our framework is based on the 3D Video SWIN
Transformer to encode both RGB and Depth modalities effectively. To address the
inherent skewness in real-world multimodal action occurrences, we propose a
training strategy using an exponentially decaying variant of the focal loss
modulating factor. Additionally, to leverage the information in both RGB and
Depth modalities, we opt for late fusion to combine the predictions from each
modality. We thoroughly evaluate our method on the action recognition task of
the MECCANO dataset, and it significantly outperforms the prior work. Notably,
our method also secured first place at the multimodal action recognition
challenge at ICIAP 2023.

我们的研究集中在认识行为，特别是来自以自我为中心视角的行为，并结合深度模态从工业环境中识别行为。针对现实世界中多模态行为发生的困难，我们提出了一种训练策略，并通过后期融合来结合来自每种模态的预测，这在 MECCANO 数据集上明显优于先前的工作，并在 ICIAP 2023 的多模态行为识别挑战中获得第一名。

工业场景中的主体视角 RGB + 深度行为识别

Egocentric RGB+Depth Action Recognition in Industry-Like Settings

Wearable cameras allow to collect images and videos of humans interacting
with the world. While human-object interactions have been thoroughly
investigated in third person vision, the problem has been understudied in
egocentric settings and in industrial scenarios. To fill this gap, we introduce
MECCANO, the first dataset of egocentric videos to study human-object
interactions in industrial-like settings. MECCANO has been acquired by 20
participants who were asked to build a motorbike model, for which they had to
interact with tiny objects and tools. The dataset has been explicitly labeled
for the task of recognizing human-object interactions from an egocentric
perspective. Specifically, each interaction has been labeled both temporally
(with action segments) and spatially (with active object bounding boxes). With
the proposed dataset, we investigate four different tasks including 1) action
recognition, 2) active object detection, 3) active object recognition and 4)
egocentric human-object interaction detection, which is a revisited version of
the standard human-object interaction detection task. Baseline results show
that the MECCANO dataset is a challenging benchmark to study egocentric
human-object interactions in industrial-like scenarios. We publicy release the
dataset at this https URL

使用可穿戴相机采集了 MECCANO 数据集，该数据集是首个研究工业场景下第一视角人物与物体交互作用的数据集，包含了四个任务，即动作识别、活动物体检测、活动物体识别和第一视角人物与物体交互作用检测。基准结果表明，MECCANO 数据集是研究工业情境下第一视角人物与物体交互作用的一项具有挑战性的基准测试。