Action recognition from an egocentric viewpoint is a crucial perception task
in robotics and enables a wide range of human-robot interactions. While most
computer vision approaches prioritize the RGB camera, the Depth modality -
which can further amplify the subtleties of actions from an egocentric
perspective - remains underexplored. Our work focuses on recognizing actions
from egocentric RGB and Depth modalities in an industry-like environment. To
study this problem, we consider the recent MECCANO dataset, which provides a
wide range of assembling actions. Our framework is based on the 3D Video SWIN
Transformer to encode both RGB and Depth modalities effectively. To address the
inherent skewness in real-world multimodal action occurrences, we propose a
training strategy using an exponentially decaying variant of the focal loss
modulating factor. Additionally, to leverage the information in both RGB and
Depth modalities, we opt for late fusion to combine the predictions from each
modality. We thoroughly evaluate our method on the action recognition task of
the MECCANO dataset, and it significantly outperforms the prior work. Notably,
our method also secured first place at the multimodal action recognition
challenge at ICIAP 2023.

我们的研究集中在认识行为，特别是来自以自我为中心视角的行为，并结合深度模态从工业环境中识别行为。针对现实世界中多模态行为发生的困难，我们提出了一种训练策略，并通过后期融合来结合来自每种模态的预测，这在 MECCANO 数据集上明显优于先前的工作，并在 ICIAP 2023 的多模态行为识别挑战中获得第一名。

工业场景中的主体视角 RGB + 深度行为识别

Egocentric RGB+Depth Action Recognition in Industry-Like Settings

Intra-class variations, distribution shifts among source and target domains
are the major challenges of category-level tasks. In this study, we address
category-level full 6D object pose estimation in the context of depth modality,
introducing a novel part-based architecture that can tackle the above-mentioned
challenges. Our architecture particularly adapts the distribution shifts
arising from shape discrepancies, and naturally removes the variations of
texture, illumination, pose, etc., so we call it as "Intrinsic Structure
Adaptor (ISA)". We engineer ISA based on the followings: i) "Semantically
Selected Centers (SSC)" are proposed in order to define the "6D pose" at the
level of categories. ii) 3D skeleton structures, which we derive as
shape-invariant features, are used to represent the parts extracted from the
instances of given categories, and privileged one-class learning is employed
based on these parts. iii) Graph matching is performed during training in such
a way that the adaptation/generalization capability of the proposed
architecture is improved across unseen instances. Experiments validate the
promising performance of the proposed architecture on both synthetic and real
datasets.

本研究介绍了一种基于部件的架构 “本质结构适配器（ISA）”，可以解决类别级别的全 6D 物体姿态估计中的巨大挑战，例如类内变化、源域和目标域之间的分布偏移，并演示了其在合成和真实数据集上的良好性能